diff --git a/.agents/skills/build-from-issue/SKILL.md b/.agents/skills/build-from-issue/SKILL.md index 4e73a61be3..c91c712b67 100644 --- a/.agents/skills/build-from-issue/SKILL.md +++ b/.agents/skills/build-from-issue/SKILL.md @@ -402,29 +402,24 @@ git diff --name-only main -- e2e/ If there are no changes under `e2e/`, skip this phase entirely. -If E2E files were modified, deploy to the local cluster and run the E2E test suite: +If E2E files were modified, run the relevant E2E lane for the driver touched by the change: ```bash -# Deploy all changes to the local k3s cluster -mise run cluster:deploy - -# Run the E2E sandbox tests -mise run test:e2e:sandbox +# Docker-backed gateway smoke E2E +mise run e2e:docker ``` -`mise run test:e2e:sandbox` depends on `cluster:deploy` and `python:proto`, then runs `uv run pytest -o python_files='test_*.py' e2e/python`. However, since the cluster may need explicit deploy for code changes beyond just E2E test files, always run `mise run cluster:deploy` first as a separate step to ensure all sandbox/proxy/policy changes are live on the cluster before running E2E tests. +Use `mise run e2e:podman`, `mise run e2e:vm`, or a Helm-backed Kubernetes E2E lane when the change targets those drivers. **E2E retry loop** (up to 3 attempts): -1. Run `mise run cluster:deploy` (only on the first attempt, or if code was changed between attempts). -2. Run `mise run test:e2e:sandbox`. -3. If tests fail: +1. Run the selected E2E lane. +2. If tests fail: - Read the pytest output carefully — identify which tests failed and why. - Distinguish between **test bugs** (the test itself is wrong) and **implementation bugs** (the code under test is wrong). - Fix the failing code or tests. - - If code changes were made (not just test fixes), re-run `mise run cluster:deploy` before retrying. - Decrement the retry counter and try again. -4. If tests pass, Phase 2 is green. +3. If tests pass, Phase 2 is green. **If all 3 E2E attempts fail**, stop and report to the user: - Which E2E tests are failing @@ -478,7 +473,6 @@ Create the PR: ```bash gh pr create \ --title "(): " \ - --assignee "@me" \ --body "$(cat <<'EOF' > **🏗️ build-from-issue-agent** @@ -577,8 +571,8 @@ Local E2E tests passed. CI does not currently run E2E tests, so this comment ser | Field | Value | |-------|-------| | **Commit** | `` | -| **Command** | `mise run test:e2e:sandbox` | -| **Cluster deploy** | `mise run cluster:deploy` (completed before test run) | +| **Command** | `` | +| **Gateway mode** | `` | | **Result** | ✅ All passed | ### Test Summary @@ -646,8 +640,9 @@ If the `state:in-progress` label is present, the skill was previously started bu | `gh pr create --title "..." --body "..."` | Create a pull request | | `gh api user --jq '.login'` | Get current GitHub username | | `mise run pre-commit` | Run pre-commit checks (includes unit tests, lint, format) | -| `mise run cluster:deploy` | Deploy all changes to local k3s cluster | -| `mise run test:e2e:sandbox` | Run E2E sandbox tests (depends on cluster:deploy) | +| `mise run e2e:docker` | Run smoke E2E against a standalone Docker-backed gateway | +| `mise run e2e:podman` | Run smoke E2E against a Podman-backed gateway | +| `mise run e2e:vm` | Run smoke E2E against the VM compute driver | ## Example Usage diff --git a/.agents/skills/create-github-issue/SKILL.md b/.agents/skills/create-github-issue/SKILL.md index 15063d2c7f..b1311a297e 100644 --- a/.agents/skills/create-github-issue/SKILL.md +++ b/.agents/skills/create-github-issue/SKILL.md @@ -114,7 +114,6 @@ GitHub built-in issue types (`Bug`, `Feature`, `Task`) should come from the matc | `--title, -t` | Issue title (required) | | `--body, -b` | Issue description | | `--label, -l` | Add label (can use multiple times) | -| `--assignee, -a` | Assign to user | | `--milestone, -m` | Add to milestone | | `--project, -p` | Add to project | | `--web` | Open in browser after creation | diff --git a/.agents/skills/create-github-pr/SKILL.md b/.agents/skills/create-github-pr/SKILL.md index 63f8dd5ab6..9050db6de5 100644 --- a/.agents/skills/create-github-pr/SKILL.md +++ b/.agents/skills/create-github-pr/SKILL.md @@ -99,22 +99,6 @@ gh pr create --title "PR title" --body "PR description" - `refactor(models): simplify deployment logic` - `chore(ci): update Python version in pipeline` -## Required PR Fields - -Every PR **must** have: - -1. **Assignee** - Always assign to yourself - -## Assignee and Reviewer - -### Always Assign to Yourself - -**Every PR must be assigned to the user creating it.** Use the `--assignee` flag: - -```bash -gh pr create --title "Title" --assignee "@me" -``` - ### Link to an Issue Use `Closes #` in the body to auto-close the issue when merged: @@ -122,7 +106,6 @@ Use `Closes #` in the body to auto-close the issue when merged: ```bash gh pr create \ --title "Fix validation error for empty requests" \ - --assignee "@me" \ --body "Closes #123 ## Summary @@ -135,7 +118,7 @@ gh pr create \ For work-in-progress that's not ready for review: ```bash -gh pr create --draft --title "WIP: New feature" --assignee "@me" +gh pr create --draft --title "WIP: New feature" ``` ### With Labels @@ -185,7 +168,6 @@ Populate the testing checklist based on what was actually run. Check boxes for s ```bash gh pr create \ --title "feat(cli): add pagination to sandbox list" \ - --assignee "@me" \ --body "$(cat <<'EOF' ## Summary @@ -222,7 +204,6 @@ EOF | ------------------- | ------------------------------------------ | | `--title, -t` | PR title (use conventional commit format) | | `--body, -b` | PR description | -| `--assignee, -a` | Assign to user (use `@me` for yourself) | | `--reviewer, -r` | Request review from user | | `--draft` | Create as draft (WIP) | | `--label, -l` | Add label (can use multiple times) | diff --git a/.agents/skills/debug-inference/SKILL.md b/.agents/skills/debug-inference/SKILL.md index 26f87b9166..6770da5987 100644 --- a/.agents/skills/debug-inference/SKILL.md +++ b/.agents/skills/debug-inference/SKILL.md @@ -174,7 +174,7 @@ openshell sandbox create -- curl https://inference.local/v1/chat/completions --j Interpretation: -- **`cluster inference is not configured`**: set the managed route with `openshell inference set` +- **`cluster inference is not configured`**: set the managed gateway route with `openshell inference set` - **`connection not allowed by policy`** on `inference.local`: unsupported method or path - **`no compatible route`**: provider type and client API shape do not match - **Connection refused / upstream unavailable / verification failures**: base URL, bind address, topology, or credentials are wrong @@ -232,7 +232,7 @@ In this case, OpenShell routing is usually working correctly. The failing hop is This is not the same issue as the Colima CoreDNS fix. -OpenShell injects `host.docker.internal` and `host.openshell.internal` into sandbox pods with `hostAliases`. That path bypasses cluster DNS lookup. If the request still times out, the usual cause is host firewall or network policy, not CoreDNS. +OpenShell injects `host.docker.internal` and `host.openshell.internal` into sandbox workloads when the selected compute platform supports it. That path bypasses runtime DNS lookup. If the request still times out, the usual cause is host firewall or network policy, not DNS. ### Verify the Problem @@ -248,40 +248,41 @@ OpenShell injects `host.docker.internal` and `host.openshell.internal` into sand curl -sS http://172.17.0.1:11434/v1/models ``` -3. Test the same endpoint from the OpenShell cluster container: +3. Test the same endpoint from a gateway or sandbox container on the Docker network: ```bash - docker exec openshell-cluster- wget -qO- -T 5 http://host.docker.internal:11434/v1/models + docker ps --filter name=openshell --format '{{.Names}}' + docker exec wget -qO- -T 5 http://host.docker.internal:11434/v1/models ``` If steps 1 and 2 succeed but step 3 times out, the host firewall or network configuration is blocking the container-to-host path. ### Fix -Allow the Docker bridge network used by the OpenShell cluster to reach the host-local inference port. The exact command depends on your firewall tooling (iptables, nftables, firewalld, UFW, etc.), but the rule should allow: +Allow the Docker bridge network used by the OpenShell gateway and sandbox containers to reach the host-local inference port. The exact command depends on your firewall tooling (iptables, nftables, firewalld, UFW, etc.), but the rule should allow: -- **Source**: the Docker bridge subnet used by the OpenShell cluster container (commonly `172.18.0.0/16`) -- **Destination**: the host gateway IP injected into sandbox pods for `host.docker.internal` (commonly `172.17.0.1`) +- **Source**: the Docker bridge subnet used by OpenShell containers (commonly `172.18.0.0/16`) +- **Destination**: the host gateway IP injected into sandbox workloads for `host.docker.internal` (commonly `172.17.0.1`) - **Port**: the inference server port (e.g. `11434/tcp` for Ollama) To find the actual values on your system: ```bash -# Docker bridge subnet for the OpenShell cluster network +# Docker bridge subnet for the OpenShell network docker network inspect $(docker network ls --filter name=openshell -q) --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}' # Host gateway IP visible from inside the container -docker exec openshell-cluster- cat /etc/hosts | grep host.docker.internal +docker exec cat /etc/hosts | grep host.docker.internal ``` Adjust the source subnet, destination IP, or port to match your local Docker network layout. ### Verify the Fix -1. Re-run the cluster container check: +1. Re-run the container network check: ```bash - docker exec openshell-cluster- wget -qO- -T 5 http://host.docker.internal:11434/v1/models + docker exec wget -qO- -T 5 http://host.docker.internal:11434/v1/models ``` 2. Re-test from a sandbox: diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index f4c5672a20..c008ee611e 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -1,417 +1,214 @@ --- name: debug-openshell-cluster -description: Debug why a openshell cluster failed to start or is unhealthy. Use when the user has a failed `openshell gateway start`, cluster health check failure, or wants to diagnose cluster infrastructure issues. Trigger keywords - debug cluster, cluster failing, cluster not starting, deploy failed, cluster troubleshoot, cluster health, cluster diagnose, why won't my cluster start, health check failed, gateway start failed, gateway not starting. +description: Debug why an OpenShell gateway deployment is unhealthy, unreachable, or unable to create sandboxes. Use when the user has a gateway health failure, Docker/Podman runtime issue, Helm install failure, Kubernetes scheduling issue, TLS secret issue, VM driver issue, or sandbox startup problem. Trigger keywords - debug gateway, gateway failing, deployment failing, helm install failing, cluster health, gateway health, gateway not starting, health check failed, sandbox pending, docker driver, podman driver, vm driver. --- -# Debug OpenShell Cluster +# Debug OpenShell Gateway Deployment -Diagnose why a openshell cluster failed to start after `openshell gateway start`. +Diagnose a gateway and its selected compute platform. Do not assume OpenShell provisions Kubernetes or runs a k3s container. OpenShell targets a reachable gateway endpoint backed by Docker, Podman, Kubernetes, or the experimental VM driver. -Use **only** `openshell` CLI commands (`openshell status`, `openshell doctor logs`, `openshell doctor exec`) to inspect and fix the cluster. Do **not** use raw `docker`, `ssh`, or `kubectl` commands directly — always go through the `openshell doctor` interface. The CLI auto-resolves local vs remote gateways, so the same commands work everywhere. +Use `openshell` first to identify the active endpoint. Then use the platform tools that match the gateway's compute driver: `docker`, `podman`, `kubectl`/`helm`, or VM driver logs. ## Overview -`openshell gateway start` creates a Docker container running k3s with the OpenShell server deployed via Helm. The deployment stages, in order, are: +The target deployment flow is: -1. **Pre-deploy check**: `openshell gateway start` in interactive mode prompts to **reuse** (keep volume, clean stale nodes) or **recreate** (destroy everything, fresh start). `mise run cluster` always recreates before deploy. -2. Ensure cluster image is available (local build or remote pull) -3. Create Docker network (`openshell-cluster`) and volume (`openshell-cluster-{name}`) -4. Create and start a privileged Docker container (`openshell-cluster-{name}`) -5. Wait for k3s to generate kubeconfig (up to 60s) -6. **Clean stale nodes**: Remove any `NotReady` k3s nodes left over from previous container instances that reused the same persistent volume -7. **Prepare local images** (if `OPENSHELL_PUSH_IMAGES` is set): In `internal` registry mode, bootstrap waits for the in-cluster registry and pushes tagged images there. In `external` mode, bootstrap uses legacy `ctr -n k8s.io images import` push-mode behavior. -8. **Reconcile TLS PKI**: Load existing TLS secrets from the cluster; if missing, incomplete, or malformed, generate fresh PKI (CA + server + client certs). Apply secrets to cluster. If rotation happened and the OpenShell workload is already running, rollout restart and wait for completion (failed rollout aborts deploy). -9. **Store CLI mTLS credentials**: Persist client cert/key/CA locally for CLI authentication. -10. Wait for cluster health checks to pass (up to 6 min): - - k3s API server readiness (`/readyz`) - - `openshell` statefulset ready in `openshell` namespace - - TLS secrets `openshell-server-tls` and `openshell-client-tls` exist in `openshell` namespace - - Sandbox supervisor binary exists at `/opt/openshell/bin/openshell-sandbox` (emits `HEALTHCHECK_MISSING_SUPERVISOR` marker if absent) +1. Operator starts or deploys the gateway. +2. Operator configures the compute driver. +3. Operator provides TLS and SSH relay material for the deployment mode. +4. The CLI registers a reachable gateway endpoint with `openshell gateway add`. +5. The gateway creates sandboxes through the selected compute driver. -For local deploys, metadata endpoint selection now depends on Docker connectivity: - -- default local Docker socket (`unix:///var/run/docker.sock`): `https://127.0.0.1:{port}` (default port 8080) -- TCP Docker daemon (`DOCKER_HOST=tcp://:`): `https://:{port}` for non-loopback hosts - -The host port is configurable via `--port` on `openshell gateway start` (default 8080) and is stored in `ClusterMetadata.gateway_port`. - -The TCP host is also added as an extra gateway TLS SAN so mTLS hostname validation succeeds. - -The default cluster name is `openshell`. The container is `openshell-cluster-{name}`. +For local evaluation only, TLS may be disabled and the gateway can be reached through `http://127.0.0.1:`. ## Prerequisites -- Docker must be running (locally or on the remote host) -- The `openshell` CLI must be available -- For remote clusters: SSH access to the remote host - -## Tools Available - -All diagnostics go through three `openshell` commands. They auto-resolve local vs remote gateways — the same commands work for both: - -```bash -# Quick connectivity check -openshell status - -# Fetch container logs -openshell doctor logs --lines 100 -openshell doctor logs --tail # stream live - -# Run any command inside the gateway container (KUBECONFIG is pre-configured) -openshell doctor exec -- kubectl get pods -A -openshell doctor exec -- kubectl -n openshell logs statefulset/openshell --tail=100 -openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml -openshell doctor exec -- df -h / -openshell doctor exec -- free -h -openshell doctor exec -- sh # interactive shell -``` +- The `openshell` CLI must be available for endpoint checks. +- Know the active gateway name and endpoint, or be able to inspect local gateway metadata. +- Know the compute platform: Docker, Podman, Kubernetes, or VM. +- For Kubernetes: `kubectl` must target the cluster that hosts OpenShell and Helm version 3 or later must be available. +- For Docker or Podman: the runtime socket must be reachable from the gateway host. ## Workflow -When the user asks to debug a cluster failure, **run diagnostics automatically** through the steps below in order. Stop and report findings as soon as a root cause is identified. Do not ask the user to choose which checks to run. - -### Determine Context - -Before running commands, establish: - -1. **Cluster name**: Default is `openshell`, giving container name `openshell-cluster-openshell` -2. **Remote or local**: The `openshell doctor` commands auto-resolve this from gateway metadata — no special flags needed for the active gateway -3. **Config directory**: `~/.config/openshell/gateways/{name}/` - -### Step 0: Quick Connectivity Check +Run diagnostics in order and stop once the root cause is clear. -Run `openshell status` first. This immediately reveals: -- Which gateway and endpoint the CLI is targeting -- Whether the CLI can reach the server (mTLS handshake success/failure) -- The server version if connected - -Common errors at this stage: -- **`tls handshake eof`**: The server isn't running or mTLS credentials are missing/mismatched -- **`connection refused`**: The container isn't running or port mapping is broken -- **`No gateway configured`**: No gateway has been deployed yet - -### Step 1: Check Container Logs - -Get recent container logs to identify startup failures: +### Step 1: Check CLI Reachability ```bash -openshell doctor logs --lines 100 +openshell gateway info +openshell status ``` -Look for: - -- DNS resolution failures in the entrypoint script -- k3s startup errors (certificate issues, port binding failures) -- Manifest copy errors from `/opt/openshell/manifests/` -- `iptables` or `cgroup` errors (privilege/capability issues) -- `Warning: br_netfilter does not appear to be loaded` — this is advisory only; many kernels work without the explicit module. Only act on it if you also see DNS failures or pod-to-service connectivity problems (see Common Failure Patterns). +Common findings: -### Step 2: Check k3s Cluster Health +- `No active gateway`: register one with `openshell gateway add `. +- Connection refused: gateway process is not running, service exposure is wrong, or a port-forward/proxy is not active. +- TLS/certificate errors: CLI mTLS bundle does not match the gateway CA, or the gateway is running with unexpected TLS settings. -Verify k3s itself is functional: +### Step 2: Identify the Compute Platform -```bash -# API server readiness -openshell doctor exec -- kubectl get --raw="/readyz" - -# Node status -openshell doctor exec -- kubectl get nodes -o wide - -# All pods -openshell doctor exec -- kubectl get pods -A -o wide -``` - -If `/readyz` fails, k3s is still starting or has crashed. Check container logs (Step 1). +Use gateway metadata, deployment values, or the user's setup notes to identify the driver. -If pods are in `CrashLoopBackOff`, `ImagePullBackOff`, or `Pending`, investigate those pods specifically. +| Platform | Primary checks | +|---|---| +| Docker | Gateway process logs, Docker daemon health, sandbox containers, image pulls. | +| Podman | Podman socket, rootless networking, sandbox containers, image pulls. | +| Kubernetes | Helm release, StatefulSet, service, secrets, sandbox pods, events. | +| VM | VM driver logs, rootfs availability, host virtualization support. | -Also check for node pressure conditions that cause the kubelet to evict pods and reject scheduling: +### Step 3: Check Docker-Backed Gateways ```bash -# Check node conditions (DiskPressure, MemoryPressure, PIDPressure) -openshell doctor exec -- kubectl get nodes -o jsonpath="{range .items[*]}{.metadata.name}{range .status.conditions[*]} {.type}={.status}{end}{\"\n\"}{end}" - -# Check disk usage inside the container -openshell doctor exec -- df -h / - -# Check memory usage -openshell doctor exec -- free -h +docker info +docker ps --filter name=openshell +docker logs --tail=200 +openshell status ``` -If any pressure condition is `True`, pods will be evicted and new ones rejected. The bootstrap now detects `HEALTHCHECK_NODE_PRESSURE` markers from the health-check script and aborts early with a clear diagnosis. To fix: free disk/memory on the host, then recreate the gateway. +Common findings: -### Step 3: Check OpenShell Server StatefulSet +- Docker daemon unavailable: start Docker Desktop or Docker Engine. +- Gateway process stopped: inspect exit status and logs. +- Sandbox image missing or pull denied: verify image reference and registry credentials. +- Sandbox never registers: check gateway logs and supervisor callback endpoint. -The OpenShell server is deployed via a HelmChart CR as a StatefulSet named `openshell` in the `openshell` namespace. Check its status: +For source checkout development, restart the local gateway with: ```bash -# StatefulSet status -openshell doctor exec -- kubectl -n openshell get statefulset/openshell -o wide - -# OpenShell pod logs -openshell doctor exec -- kubectl -n openshell logs statefulset/openshell --tail=100 - -# Describe statefulset for events -openshell doctor exec -- kubectl -n openshell describe statefulset/openshell - -# Helm install job logs (the job that installs the OpenShell chart) -openshell doctor exec -- kubectl -n kube-system logs -l job-name=helm-install-openshell --tail=200 +mise run gateway:docker ``` -Common issues: - -- **Replicas 0/0**: The StatefulSet has been scaled to zero — no pods are running. This can happen after a failed deploy, manual scale-down, or Helm values misconfiguration. Fix: `openshell doctor exec -- kubectl -n openshell scale statefulset openshell --replicas=1` -- **ImagePullBackOff**: The component image failed to pull. In `internal` mode, verify internal registry readiness and pushed image tags (Step 5). In `external` mode, check `/etc/rancher/k3s/registries.yaml` credentials/endpoints and DNS (Step 8). Default external registry is `ghcr.io/nvidia/openshell/` (public, no auth required). If using a private registry, ensure `--registry-username` and `--registry-token` (or `OPENSHELL_REGISTRY_USERNAME`/`OPENSHELL_REGISTRY_TOKEN`) were provided during deploy. -- **CrashLoopBackOff**: The server is crashing. Check pod logs for the actual error. -- **Pending**: Insufficient resources or scheduling constraints. - -### Step 4: Check Networking - -The OpenShell server is exposed via a NodePort service on port `30051`: +### Step 4: Check Podman-Backed Gateways ```bash -# Service status -openshell doctor exec -- kubectl -n openshell get service/openshell +podman info +podman ps --filter name=openshell +podman logs --tail=200 +openshell status ``` -Expected port: `30051/tcp` (mapped to configurable host port, default 8080; set via `--port` on deploy). +Common findings: -### Step 5: Check Image Availability +- Podman socket unavailable: start or expose the user socket. +- Rootless networking unavailable: inspect Podman network configuration. +- Sandbox image missing or pull denied: verify image reference and registry credentials. +- Supervisor cannot call back: check callback endpoint and gateway logs. -Component images (server, sandbox) can reach kubelet via two paths: - -**Local/external pull mode** (default local via `mise run cluster`): Local images are tagged to the configured local registry base (default `127.0.0.1:5000/openshell/*`), pushed to that registry, and pulled by k3s via `registries.yaml` mirror endpoint (typically `host.docker.internal:5000`). The `cluster` task pushes prebuilt local tags (`openshell/*:dev`, falling back to `localhost:5000/openshell/*:dev` or `127.0.0.1:5000/openshell/*:dev`). - -Gateway image builds now stage a partial Rust workspace from `deploy/docker/Dockerfile.images`. If cargo fails with a missing manifest under `/build/crates/...`, verify that every current gateway dependency crate (including `openshell-driver-kubernetes`) is copied into the staged workspace there. +### Step 5: Check Kubernetes Helm Gateways ```bash -# Verify image refs currently used by openshell deployment -openshell doctor exec -- kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}" - -# Verify registry mirror/auth endpoint configuration -openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml +helm -n openshell status openshell +helm -n openshell get values openshell +kubectl -n openshell get statefulset,pod,svc,pvc +kubectl -n openshell logs statefulset/openshell --tail=200 +kubectl -n openshell rollout status statefulset/openshell ``` -**Legacy push mode**: Images are imported into the k3s containerd `k8s.io` namespace. +Look for failed installs, unexpected values, missing namespace, wrong image tag, TLS settings that do not match the registered endpoint, and scheduling failures. -```bash -# Check if images were imported into containerd (k3s default namespace is k8s.io) -openshell doctor exec -- ctr -a /run/k3s/containerd/containerd.sock images ls | grep openshell -``` - -**External pull mode** (remote deploy, or local with `OPENSHELL_REGISTRY_HOST`/`IMAGE_REPO_BASE` pointing at a non-local registry): Images are pulled from an external registry at runtime. The entrypoint generates `/etc/rancher/k3s/registries.yaml`. +Check required Helm deployment secrets: ```bash -# Verify registries.yaml exists and has credentials -openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml - -# Test pulling an image manually from inside the cluster -openshell doctor exec -- crictl pull ghcr.io/nvidia/openshell/gateway:latest +kubectl -n openshell get secret \ + openshell-ssh-handshake \ + openshell-server-tls \ + openshell-server-client-ca \ + openshell-client-tls ``` -If `registries.yaml` is missing or has wrong values, verify env wiring (`OPENSHELL_REGISTRY_HOST`, `OPENSHELL_REGISTRY_INSECURE`, username/password for authenticated registries). - -### Step 6: Check mTLS / PKI - -TLS certificates are generated by the `openshell-bootstrap` crate (using `rcgen`) and stored as K8s secrets before the Helm release installs. There is no PKI job or cert-manager — certificates are applied directly via `kubectl apply`. +Check the image references currently used by the gateway deployment: ```bash -# Check if the three TLS secrets exist -openshell doctor exec -- kubectl -n openshell get secret openshell-server-tls openshell-server-client-ca openshell-client-tls - -# Inspect server cert expiry (if openssl is available in the container) -openshell doctor exec -- sh -c 'kubectl -n openshell get secret openshell-server-tls -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -noout -dates 2>/dev/null || echo "openssl not available"' - -# Check if CLI-side mTLS files exist locally -ls -la ~/.config/openshell/gateways//mtls/ +kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" +helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage' ``` -On redeploy, bootstrap reuses existing secrets if they are valid PEM. If secrets are missing or malformed, fresh PKI is generated and the OpenShell workload is automatically restarted. If the rollout restart fails after rotation, the deploy aborts and CLI-side certs are not updated. Certificates use rcgen defaults (effectively never expire). +The gateway image and `server.supervisorImage` should use the same build tag in branch and E2E deploys. A stale supervisor image can make sandbox behavior lag behind gateway policy or proto changes. -If the local mTLS files are missing but the secrets exist in the cluster, you can extract them manually: +For plaintext local evaluation, confirm the chart has: ```bash -mkdir -p ~/.config/openshell/gateways//mtls -openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ~/.config/openshell/gateways//mtls/ca.crt -openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways//mtls/tls.crt -openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways//mtls/tls.key +helm -n openshell get values openshell | grep -E 'disableTls|grpcEndpoint' ``` -Common mTLS issues: -- **Secrets missing**: The `openshell` namespace may not have been created yet (Helm controller race). Bootstrap waits up to 2 minutes for the namespace. -- **mTLS mismatch after manual secret deletion**: Delete all three secrets and redeploy — bootstrap will regenerate and restart the workload. -- **CLI can't connect after redeploy**: Check that `~/.config/openshell/gateways//mtls/` contains `ca.crt`, `tls.crt`, `tls.key` and that they were updated at deploy time. -- **Local mTLS files missing**: The gateway was deployed but CLI credentials weren't persisted (e.g., interrupted deploy). Extract from the cluster secret as shown above. +Expected shape: -### Step 7: Check Kubernetes Events +```yaml +server: + disableTls: true + grpcEndpoint: http://openshell.openshell.svc.cluster.local:8080 +``` -Events catch scheduling failures, image pull errors, and resource issues: +Check service exposure: ```bash -openshell doctor exec -- kubectl get events -A --sort-by=.lastTimestamp | tail -n 50 +kubectl -n openshell get svc openshell -o wide +kubectl -n openshell get endpoints openshell ``` -Look for: - -- `FailedScheduling` — resource constraints -- `ImagePullBackOff` / `ErrImagePull` — registry auth failure or DNS issue (check `/etc/rancher/k3s/registries.yaml`) -- `CrashLoopBackOff` — application crashes -- `OOMKilled` — memory limits too low -- `FailedMount` — volume issues - -### Step 8: Check GPU Device Plugin and CDI (GPU gateways only) - -Skip this step for non-GPU gateways. - -The NVIDIA device plugin DaemonSet must be running and healthy before GPU sandboxes can be created. It uses CDI injection (`deviceListStrategy: cdi-cri`) to inject GPU devices into sandbox pods — no `runtimeClassName` is set on sandbox pods. +For local port-forward testing: ```bash -# DaemonSet status — numberReady must be >= 1 -openshell doctor exec -- kubectl get daemonset -n nvidia-device-plugin - -# Device plugin pod logs — look for "CDI" lines confirming CDI mode is active -openshell doctor exec -- kubectl logs -n nvidia-device-plugin -l app.kubernetes.io/name=nvidia-device-plugin --tail=50 - -# List CDI devices registered by the device plugin (requires nvidia-ctk in the cluster image). -# Device plugin CDI entries use the vendor string "k8s.device-plugin.nvidia.com" so entries -# will be prefixed "k8s.device-plugin.nvidia.com/gpu=". If the list is empty, CDI spec -# generation has not completed yet. -openshell doctor exec -- nvidia-ctk cdi list - -# Verify CDI spec files were generated on the node -openshell doctor exec -- ls /var/run/cdi/ - -# Helm install job logs for the device plugin chart -openshell doctor exec -- kubectl -n kube-system logs -l job-name=helm-install-nvidia-device-plugin --tail=100 - -# Confirm a GPU sandbox pod has no runtimeClassName (CDI injection, not runtime class) -openshell doctor exec -- kubectl get pod -n openshell -o jsonpath='{range .items[*]}{.metadata.name}{" runtimeClassName="}{.spec.runtimeClassName}{"\n"}{end}' +kubectl -n openshell port-forward svc/openshell 8080:8080 +openshell gateway add http://127.0.0.1:8080 --local --name local +openshell status ``` -Common issues: - -- **DaemonSet 0/N ready**: The device plugin chart may still be deploying (k3s Helm controller can take 1–2 min) or the pod is crashing. Check pod logs. -- **`nvidia-ctk cdi list` returns no `k8s.device-plugin.nvidia.com/gpu=` entries**: CDI spec generation has not completed. The device plugin may still be starting or the `cdi-cri` strategy isn't active. Verify `deviceListStrategy: cdi-cri` is in the rendered Helm values. -- **No CDI spec files at `/var/run/cdi/`**: Same as above — device plugin hasn't written CDI specs yet. -- **`HEALTHCHECK_GPU_DEVICE_PLUGIN_NOT_READY` in health check logs**: Device plugin has no ready pods. Check DaemonSet events and pod logs. - -### Step 9: Check DNS Resolution - -DNS misconfiguration is a common root cause, especially on remote/Linux hosts: +If the gateway is healthy but sandbox creation fails: ```bash -# Check the resolv.conf k3s is using -openshell doctor exec -- cat /etc/rancher/k3s/resolv.conf - -# Test DNS resolution from inside the container -openshell doctor exec -- sh -c 'nslookup google.com || wget -q -O /dev/null http://google.com && echo "network ok" || echo "network unreachable"' +kubectl -n openshell get pods +kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50 +kubectl -n openshell logs statefulset/openshell --tail=200 ``` -Check the entrypoint's DNS decision in the container logs: +Check the configured sandbox namespace: ```bash -openshell doctor logs --lines 20 +helm -n openshell get values openshell | grep sandboxNamespace ``` -The entrypoint script selects DNS resolvers in this priority: +Then inspect sandbox resources in that namespace. -1. Viable nameservers from `/etc/resolv.conf` (not loopback/link-local) -2. Docker `ExtServers` from `/etc/resolv.conf` comments -3. Host gateway IP (Docker Desktop only, `192.168.*`) -4. Fallback to `8.8.8.8` / `8.8.4.4` +### Step 6: Check VM-Backed Gateways -If DNS is broken, all image pulls from the distribution registry will fail, as will pods that need external network access. +Use the VM driver logs and host diagnostics available in the user's environment. Verify: -## Common Failure Patterns +- The VM driver process is running and reachable by the gateway. +- The runtime rootfs exists and matches the expected architecture. +- Host virtualization support is enabled. +- The sandbox supervisor can establish its callback connection to the gateway. -| Symptom | Likely Cause | Fix | -|---------|-------------|-----| -| `tls handshake eof` from `openshell status` | Server not running or mTLS credentials missing/mismatched | Check StatefulSet replicas (Step 3) and mTLS files (Step 6) | -| StatefulSet `0/0` replicas | StatefulSet scaled to zero (failed deploy, manual scale-down, or Helm misconfiguration) | `openshell doctor exec -- kubectl -n openshell scale statefulset openshell --replicas=1` | -| Local mTLS files missing | Deploy was interrupted before credentials were persisted | Extract from cluster secret `openshell-client-tls` (Step 6) | -| Container not found | Image not built | `mise run docker:build:cluster` (local) or re-deploy (remote) | -| Container exited, OOMKilled | Insufficient memory | Increase host memory or reduce workload | -| Container exited, non-zero exit | k3s crash, port conflict, privilege issue | Check `openshell doctor logs` for details | -| `/readyz` fails | k3s still starting or crashed | Wait longer or check container logs for k3s errors | -| OpenShell pods `Pending` | Insufficient CPU/memory for scheduling, or PVC not bound | `openshell doctor exec -- kubectl describe pod -n openshell` and `openshell doctor exec -- kubectl get pvc -n openshell` | -| OpenShell pods `CrashLoopBackOff` | Server application error | `openshell doctor exec -- kubectl -n openshell logs statefulset/openshell` | -| OpenShell pods `ImagePullBackOff` (push mode) | Images not imported or wrong containerd namespace | Check `openshell doctor exec -- ctr -a /run/k3s/containerd/containerd.sock -n k8s.io images ls` (Step 5) | -| OpenShell pods `ImagePullBackOff` (pull mode) | Registry auth or DNS issue | Check `openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml` and DNS (Step 8) | -| Image import fails | Corrupt tar stream or containerd not ready | Retry after k3s is fully started; check container logs | -| Push mode images not found by kubelet | Imported into wrong containerd namespace | Must use `k3s ctr -n k8s.io images import`, not `k3s ctr images import` | -| mTLS secrets missing | Bootstrap couldn't apply secrets (namespace not ready) | Check deploy logs and verify `openshell` namespace exists (Step 6) | -| mTLS mismatch after redeploy | PKI rotated but workload not restarted, or rollout failed | Check that all three TLS secrets exist and that the openshell pod restarted after cert rotation (Step 6) | -| Helm install job failed | Chart values error or dependency issue | `openshell doctor exec -- kubectl -n kube-system logs -l job-name=helm-install-openshell` | -| NFD/GFD DaemonSets present (`node-feature-discovery`, `gpu-feature-discovery`) | Cluster was deployed before NFD/GFD were disabled (pre-simplify-device-plugin change) | These are harmless but add overhead. Clean up: `openshell doctor exec -- kubectl delete daemonset -n nvidia-device-plugin -l app.kubernetes.io/name=node-feature-discovery` and similarly for GFD. The `nvidia.com/gpu.present` node label is no longer applied; device plugin scheduling no longer requires it. | -| Architecture mismatch (remote) | Built on arm64, deploying to amd64 | Cross-build the image for the target architecture | -| Port conflict | Another service on the configured gateway host port (default 8080) | Stop conflicting service or use `--port` on `openshell gateway start` to pick a different host port | -| gRPC connect refused to `127.0.0.1:443` in CI | Docker daemon is remote (`DOCKER_HOST=tcp://...`) but metadata still points to loopback | Verify metadata endpoint host matches `DOCKER_HOST` and includes non-loopback host | -| DNS failures inside container | Entrypoint DNS detection failed | `openshell doctor exec -- cat /etc/rancher/k3s/resolv.conf` and `openshell doctor logs --lines 20` | -| Pods can't reach kube-dns / ClusterIP services | `br_netfilter` not loaded; bridge traffic bypasses iptables DNAT rules | `sudo modprobe br_netfilter` on the host, then `echo br_netfilter \| sudo tee /etc/modules-load.d/br_netfilter.conf` to persist. Known to be required on Jetson Linux 5.15-tegra; other kernels (e.g. standard x86/aarch64 Linux) may have bridge netfilter built in and work without the module. The entrypoint logs a warning when `/proc/sys/net/bridge/bridge-nf-call-iptables` is absent but does not abort — only act on it if DNS or service connectivity is actually broken. | -| Node DiskPressure / MemoryPressure / PIDPressure | Insufficient disk, memory, or PIDs on host | Free disk (`docker system prune -a --volumes`), increase memory, or expand host resources. Bootstrap auto-detects via `HEALTHCHECK_NODE_PRESSURE` marker | -| Pods evicted with "The node had condition: [DiskPressure]" | Host disk full, kubelet evicting pods | Free disk space on host, then `openshell gateway destroy && openshell gateway start` | -| `metrics-server` errors in logs | Normal k3s noise, not the root cause | These errors are benign — look for the actual failing health check component | -| Stale NotReady nodes from previous deploys | Volume reused across container recreations | The deploy flow now auto-cleans stale nodes; if it still fails, manually delete NotReady nodes (see Step 2) or choose "Recreate" when prompted | -| gRPC `UNIMPLEMENTED` for newer RPCs in push mode | Helm values still point at older pulled images instead of the pushed refs | Verify rendered `openshell-helmchart.yaml` uses the expected push refs (`server`, `sandbox`, `pki-job`) and not `:latest` | -| Sandbox pods crash with `/opt/openshell/bin/openshell-sandbox: no such file or directory` | Supervisor binary missing from cluster image | The cluster image was built/published without the `supervisor-builder` target in `deploy/docker/Dockerfile.images`. Rebuild with `mise run docker:build:cluster` and recreate gateway. Bootstrap auto-detects via `HEALTHCHECK_MISSING_SUPERVISOR` marker | -| `HEALTHCHECK_MISSING_SUPERVISOR` in health check logs | `/opt/openshell/bin/openshell-sandbox` not found in gateway container | Rebuild cluster image: `mise run docker:build:cluster`, then `openshell gateway destroy && openshell gateway start` | -| `nvidia-ctk cdi list` returns no `k8s.device-plugin.nvidia.com/gpu=` entries | CDI specs not yet generated by device plugin | Device plugin may still be starting; wait and retry, or check pod logs (Step 8) | - -## Full Diagnostic Dump - -Run all diagnostics at once for a comprehensive report: +Then run: ```bash -echo "=== Connectivity Check ===" openshell status +openshell logs +``` -echo "=== Container Logs (last 50 lines) ===" -openshell doctor logs --lines 50 - -echo "=== k3s Readiness ===" -openshell doctor exec -- kubectl get --raw='/readyz' - -echo "=== Nodes ===" -openshell doctor exec -- kubectl get nodes -o wide - -echo "=== Node Conditions ===" -openshell doctor exec -- kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{range .status.conditions[*]} {.type}={.status}{end}{"\n"}{end}' - -echo "=== Disk Usage ===" -openshell doctor exec -- df -h / - -echo "=== All Pods ===" -openshell doctor exec -- kubectl get pods -A -o wide - -echo "=== Failing Pods ===" -openshell doctor exec -- kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded - -echo "=== OpenShell StatefulSet ===" -openshell doctor exec -- kubectl -n openshell get statefulset/openshell -o wide - -echo "=== OpenShell Service ===" -openshell doctor exec -- kubectl -n openshell get service/openshell - -echo "=== TLS Secrets ===" -openshell doctor exec -- kubectl -n openshell get secret openshell-server-tls openshell-server-client-ca openshell-client-tls - -echo "=== Recent Events ===" -openshell doctor exec -- kubectl get events -A --sort-by=.lastTimestamp | tail -n 50 - -echo "=== Helm Install OpenShell Logs ===" -openshell doctor exec -- kubectl -n kube-system logs -l job-name=helm-install-openshell --tail=100 - -echo "=== Registry Configuration ===" -openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml - -echo "=== Supervisor Binary ===" -openshell doctor exec -- ls -la /opt/openshell/bin/openshell-sandbox - -echo "=== DNS Configuration ===" -openshell doctor exec -- cat /etc/rancher/k3s/resolv.conf +## Common Failure Patterns -# GPU gateways only -echo "=== GPU Device Plugin ===" -openshell doctor exec -- kubectl get daemonset -n nvidia-device-plugin -openshell doctor exec -- nvidia-ctk cdi list -``` +| Symptom | Likely cause | Check | +|---|---|---| +| `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs | +| Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs | +| Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs | +| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod ` | +| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs statefulset/openshell` | +| CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways//mtls/` | +| Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials | +| `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was auto-applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Confirm the cluster image only bundles core manifests; apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only when `grpcRoute` is enabled | + +## Reporting + +When handing results back to the user, include: + +- Active gateway endpoint and auth mode. +- Compute platform and driver. +- Gateway process or workload status. +- Recent gateway log summary. +- Missing or malformed TLS or SSH relay material. +- Service exposure status. +- Sandbox workload status. +- The exact command that failed and the shortest fix. diff --git a/.agents/skills/fix-security-issue/SKILL.md b/.agents/skills/fix-security-issue/SKILL.md index 7df9b4178c..22ffd42541 100644 --- a/.agents/skills/fix-security-issue/SKILL.md +++ b/.agents/skills/fix-security-issue/SKILL.md @@ -207,7 +207,6 @@ Create a PR that closes the security issue. Put the full fix summary in the PR d ```bash gh pr create \ --title "fix(security): " \ - --assignee "@me" \ --label "topic:security" \ --body "$(cat <<'EOF' > **🔧 security-fix-agent** diff --git a/.agents/skills/generate-sandbox-policy/SKILL.md b/.agents/skills/generate-sandbox-policy/SKILL.md index 7149bbd610..95767fe3fb 100644 --- a/.agents/skills/generate-sandbox-policy/SKILL.md +++ b/.agents/skills/generate-sandbox-policy/SKILL.md @@ -439,7 +439,7 @@ network_policies: # ``` -The `filesystem_policy`, `landlock`, and `process` sections above are sensible defaults. Tell the user these are defaults and may need adjustment for their environment. Cluster inference is configured separately through `openshell cluster inference set/get`. The generated `network_policies` block is the primary output. +The `filesystem_policy`, `landlock`, and `process` sections above are sensible defaults. Tell the user these are defaults and may need adjustment for their environment. Gateway inference is configured separately through `openshell inference set/get`. The generated `network_policies` block is the primary output. If the user provides a file path, write to it. Otherwise, ask where to place it. A common convention is a project-local policy file (e.g., `sandbox-policy.yaml`) passed to `openshell sandbox create --policy ` or set via the `OPENSHELL_SANDBOX_POLICY` env var. diff --git a/.agents/skills/generate-sandbox-policy/examples.md b/.agents/skills/generate-sandbox-policy/examples.md index ede5111343..b6acbee8bf 100644 --- a/.agents/skills/generate-sandbox-policy/examples.md +++ b/.agents/skills/generate-sandbox-policy/examples.md @@ -858,7 +858,7 @@ network_policies: - { path: /usr/local/bin/claude } ``` -The agent notes that `filesystem_policy`, `landlock`, and `process` are sensible defaults that may need adjustment, and that cluster inference is configured separately via `openshell cluster inference set/get` rather than an `inference` policy block. +The agent notes that `filesystem_policy`, `landlock`, and `process` are sensible defaults that may need adjustment, and that gateway inference is configured separately via `openshell inference set/get` rather than an `inference` policy block. --- diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md new file mode 100644 index 0000000000..986ce14905 --- /dev/null +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -0,0 +1,200 @@ +--- +name: helm-dev-environment +description: Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local. +--- + +# Helm Dev Environment + +Set up, run, and tear down the local Kubernetes development environment for OpenShell. +The stack is: **k3d** (Docker-backed k3s) for the cluster, **Skaffold** for image builds and Helm deploys, and the **OpenShell Helm chart** (`deploy/helm/openshell/`). + +--- + +## Prerequisites + +- Docker Desktop (macOS) or Docker Engine (Linux) running +- `mise install` completed (provides `k3d`, `kubectl`, `skaffold`, `helm`) + +--- + +## Startup + +### 1. Create the cluster + +```bash +mise run helm:k3s:create +``` + +Creates a k3d cluster and merges its kubeconfig into the worktree-local `kubeconfig` file. +Also applies base manifests (`deploy/kube/manifests/agent-sandbox.yaml`). Traefik is +disabled at cluster creation time. + +**Multi-worktree support:** the cluster name is derived from the last component of the +current git branch (e.g. branch `kube-support/local-dev/tmutch` → cluster +`openshell-dev-tmutch`). Each worktree therefore gets its own isolated cluster and its +own `kubeconfig` file. Override with `HELM_K3S_CLUSTER_NAME` to force a specific name +or share one cluster across worktrees. + +Port mappings created at cluster time (cannot be changed without recreating): + +| Host port | Target | Used by | +|-----------|--------|---------| +| `8080` | Port `80` via k3d load balancer | Envoy Gateway LoadBalancer service (`values-gateway.yaml`) | + +Override with env vars before running `helm:k3s:create`: +- `HELM_K3S_LB_HOST_PORT` (default: `8080`) + +### 2. Deploy OpenShell + +**Iterative dev** (rebuilds on file changes, recommended during active development): +```bash +mise run helm:skaffold:dev +``` + +**One-shot deploy** (build once and leave running): +```bash +mise run helm:skaffold:run +``` + +Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm +chart. The `pkiInitJob` hook runs on first install to generate mTLS secrets. Envoy Gateway opt-in; see the Optional Add-ons section below. + +The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or `kubectl port-forward`. + +### TLS behaviour + +`values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run +plaintext by default. To test with TLS enabled, comment out that line and redeploy. + +| Mode | `server.disableTls` | Gateway scheme | +|------|---------------------|----------------| +| Skaffold dev (default) | `true` | `http://` | +| TLS enabled | `false` (or omitted) | `https://` | + +### Connecting via port-forward + +Port `8080` is already bound by the k3d load balancer when Envoy Gateway is active, so +the port-forward uses local port `8090` to avoid a collision: + +```bash +KUBECONFIG=kubeconfig kubectl port-forward -n openshell svc/openshell 8090:8080 +``` + +**Plaintext (default Skaffold deploy):** + +```bash +openshell sandbox list --gateway-endpoint http://localhost:8090 +``` + +**With mTLS enabled** — extract the client cert the PKI hook wrote to the cluster, +then place it where the CLI expects it. Run once after each fresh install: + +```bash +mkdir -p ~/.config/openshell/gateways/openshell/mtls +KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \ + -o jsonpath='{.data.ca\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt +KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \ + -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt +KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \ + -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key +``` + +The server cert SANs include `localhost` and `127.0.0.1`, so hostname verification +passes over a port-forward without any extra flags: + +```bash +openshell sandbox list --gateway-endpoint https://localhost:8090 +``` + +--- + +## Teardown + +### Remove the Helm releases (keep cluster) + +```bash +mise run helm:skaffold:delete +``` + +### Delete the cluster entirely + +```bash +mise run helm:k3s:delete +``` + +This removes the k3d cluster and all resources. Kubeconfig context is left behind +but will point to a deleted cluster — safe to ignore or clean up manually. + +--- + +## Optional Add-ons + +Each add-on requires uncommenting the corresponding `valuesFiles` entry in +`deploy/helm/openshell/skaffold.yaml` before running `helm:skaffold:dev` or `helm:skaffold:run`. + +### Envoy Gateway (Gateway API / GRPCRoute) + +Envoy Gateway is already installed by Skaffold (the `envoy-gateway` Helm release in +`skaffold.yaml`). To activate routing: + +1. Uncomment `#- values-gateway.yaml` in `skaffold.yaml` +2. Redeploy: `mise run helm:skaffold:run` +3. Apply the GatewayClass: `mise run helm:gateway:apply` +4. Access: `http://127.0.0.1:8080` + +`values-gateway.yaml` creates a `Gateway` (listener on port 80, class `eg`) and a +`GRPCRoute` in the `openshell` namespace. Envoy Gateway provisions a LoadBalancer +service for the proxy; klipper-lb binds it to hostPort 80, reachable via the +`8080:80` load balancer port mapping. + +### Keycloak OIDC + +One-time setup — only needed once per cluster lifetime: + +```bash +mise run keycloak:k8s:setup +``` + +This deploys Keycloak (`quay.io/keycloak/keycloak:24.0`) into the `keycloak` namespace, +imports the openshell realm from `scripts/keycloak-realm.json`, and prints a port-forward +command for acquiring tokens from the CLI. + +Then activate OIDC in the OpenShell Helm chart: +1. Uncomment `#- values-keycloak.yaml` in `skaffold.yaml` +2. Redeploy: `mise run helm:skaffold:run` + +To remove Keycloak: +```bash +mise run keycloak:k8s:teardown +``` + +--- + +## Cluster Lifecycle (suspend/resume) + +Stop the cluster without losing state (faster than delete/recreate): +```bash +mise run helm:k3s:stop +mise run helm:k3s:start +``` + +Check cluster status: +```bash +mise run helm:k3s:status +``` + +--- + +## Key Files + +| Path | Purpose | +|------|---------| +| `deploy/helm/openshell/skaffold.yaml` | Skaffold config — images, Helm releases, values overlays | +| `deploy/helm/openshell/values.yaml` | Default Helm values | +| `deploy/helm/openshell/values-skaffold.yaml` | Dev overrides (image pull policy, local image names) | +| `deploy/helm/openshell/values-cert-manager.yaml` | cert-manager TLS overlay (opt-in; disables pkiInitJob) | +| `deploy/helm/openshell/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay | +| `deploy/helm/openshell/values-keycloak.yaml` | Keycloak OIDC overlay | +| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass for Envoy Gateway (`mise run helm:gateway:apply`) | +| `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status | +| `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy + realm import | diff --git a/.agents/skills/openshell-cli/SKILL.md b/.agents/skills/openshell-cli/SKILL.md index 132c996861..39af09e082 100644 --- a/.agents/skills/openshell-cli/SKILL.md +++ b/.agents/skills/openshell-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: openshell-cli -description: Guide agents through using the OpenShell CLI (openshell) for sandbox management, provider configuration, policy iteration, BYOC workflows, and inference routing. Covers basic through advanced multi-step workflows. Trigger keywords - openshell, sandbox create, sandbox connect, logs, provider create, policy set, policy get, image push, forward, port forward, BYOC, bring your own container, use openshell, run openshell, CLI usage, manage sandbox, manage provider, gateway start, gateway select. +description: Guide agents through using the OpenShell CLI (openshell) for sandbox management, gateway registration, provider configuration, policy iteration, BYOC workflows, and inference routing. Covers basic through advanced multi-step workflows. Trigger keywords - openshell, sandbox create, sandbox connect, logs, provider create, policy set, policy get, image push, forward, port forward, BYOC, bring your own container, use openshell, run openshell, CLI usage, manage sandbox, manage provider, gateway add, gateway select. --- # OpenShell CLI @@ -26,8 +26,9 @@ This is your primary fallback. Use it freely -- the CLI's help output is authori ## Prerequisites - `openshell` is on the PATH (install via `cargo install --path crates/openshell-cli`) -- Docker is running (required for gateway operations and BYOC) -- For remote clusters: SSH access to the target host +- A reachable OpenShell gateway backed by Docker, Podman, Kubernetes, or the experimental VM driver +- Docker is running only when using BYOC local builds or Docker-backed development workflows +- For Kubernetes deployments: `kubectl` and Helm access to the target cluster ## Command Reference @@ -37,29 +38,23 @@ See [cli-reference.md](cli-reference.md) for the full command tree with all flag ## Workflow 1: Getting Started -Use this workflow when no cluster exists yet and the user wants to get a sandbox running for the first time. +Use this workflow when the user has a gateway endpoint and wants to get a sandbox running for the first time. -### Step 1: Bootstrap a cluster +### Step 1: Register a gateway ```bash -openshell gateway start +openshell gateway add http://127.0.0.1:8080 --local --name local ``` -This provisions a local k3s cluster in Docker. The CLI will prompt interactively if a cluster already exists. The cluster is automatically set as the active gateway. +Use an `http://` endpoint only for trusted local port-forwarding or a protected private path. For a gateway behind an authenticated reverse proxy, register its HTTPS endpoint with `openshell gateway add https://gateway.example.com`. -For remote deployment: - -```bash -openshell gateway start --remote user@host --ssh-key ~/.ssh/id_rsa -``` - -### Step 2: Verify the cluster +### Step 2: Verify the gateway ```bash openshell status ``` -Confirm the cluster is reachable and shows a version. +Confirm the gateway is reachable and shows a version. ### Step 3: Create a sandbox @@ -69,7 +64,7 @@ The simplest way to get a sandbox running: openshell sandbox create ``` -This creates a sandbox with defaults and drops you into an interactive shell. The CLI auto-bootstraps a cluster if none exists. +This creates a sandbox with defaults and drops you into an interactive shell. **Shortcut for known tools**: When the trailing command is a recognized tool, the CLI auto-creates the required provider from local credentials: @@ -325,7 +320,7 @@ openshell sandbox create --from ./Dockerfile --name my-app The `--from` flag accepts a Dockerfile path, a directory containing a Dockerfile, a full image reference (e.g. `myregistry.com/img:tag`), or a community sandbox name (e.g. `openclaw`). -When given a Dockerfile or directory, the image is built locally via Docker and imported directly into the cluster's containerd runtime. No external registry needed. +When given a Dockerfile or directory, the image is built locally via Docker and delivered through the selected compute driver. Docker and Podman-backed gateways can use local images directly. Kubernetes gateways usually need the image available to the cluster through a registry or driver-supported image push path. When `--from` is specified, the CLI: - Clears default `run_as_user`/`run_as_group` (custom images may not have the `sandbox` user) @@ -421,10 +416,14 @@ Watch for `deny` actions that indicate the user's work is being blocked by polic When denied actions are observed: -1. Pull current policy: `openshell policy get work-session --full > policy.yaml` -2. Modify the policy to allow the blocked actions (use `generate-sandbox-policy` skill for content) -3. Push the update: `openshell policy set work-session --policy policy.yaml --wait` -4. Verify: `openshell policy list work-session` +1. Prefer incremental updates for additive network changes: + `openshell policy update work-session --add-endpoint api.github.com:443:read-only:rest:enforce --binary /usr/bin/gh --wait` + `openshell policy update work-session --add-allow 'api.github.com:443:POST:/repos/*/issues' --wait` +2. Use full YAML replacement when the change is broad or touches non-network fields: + `openshell policy get work-session --full > policy.yaml` + Modify the policy to allow the blocked actions (use `generate-sandbox-policy` skill for content) + `openshell policy set work-session --policy policy.yaml --wait` +3. Verify: `openshell policy list work-session` The user does not need to disconnect -- policy updates are hot-reloaded within ~30 seconds (or immediately when using `--wait`, which polls for confirmation). @@ -467,8 +466,8 @@ openshell inference get ### How sandboxes use it - Agents send HTTPS requests to `inference.local`. -- The sandbox intercepts those requests locally and routes them through the cluster inference config. -- Sandbox policy is separate from cluster inference configuration. +- The sandbox intercepts those requests locally and routes them through the gateway inference config. +- Sandbox policy is separate from gateway inference configuration. --- @@ -478,35 +477,29 @@ openshell inference get ```bash openshell gateway select # See all gateways (no args shows list) -openshell gateway select my-cluster # Switch active gateway +openshell gateway select production # Switch active gateway openshell status # Verify connectivity ``` -### Lifecycle +### Registration ```bash -openshell gateway start # Start local cluster -openshell gateway stop # Stop (preserves state) -openshell gateway start # Restart (reuses state) -openshell gateway destroy # Destroy permanently +openshell gateway add http://127.0.0.1:8080 --local --name local +openshell gateway add https://gateway.example.com --name production +openshell gateway destroy --name local # Remove local registration ``` -### Remote clusters +### Platform-specific deployment inspection ```bash -# Deploy to remote host -openshell gateway start --remote user@host --ssh-key ~/.ssh/id_rsa --name remote-cluster - -# View gateway container logs -openshell doctor logs --name remote-cluster - -# Run kubectl inside the remote gateway container -openshell doctor exec --name remote-cluster -- kubectl get pods -A - -# Get cluster info -openshell gateway info --name remote-cluster +# Inspect a Kubernetes Helm release and gateway pod +helm -n openshell status openshell +kubectl -n openshell get pods,svc +kubectl -n openshell logs statefulset/openshell --tail=100 ``` +For Docker, Podman, and VM-backed gateways, inspect the gateway process or container logs and the selected runtime directly. + --- ## Self-Teaching via `--help` @@ -535,14 +528,15 @@ $ openshell sandbox upload --help | Task | Command | |------|---------| -| Deploy local cluster | `openshell gateway start` | -| Check cluster health | `openshell status` | +| Register local port-forwarded gateway | `openshell gateway add http://127.0.0.1:8080 --local --name local` | +| Check gateway health | `openshell status` | | List/switch gateways | `openshell gateway select [name]` | | Create sandbox (interactive) | `openshell sandbox create` | | Create sandbox with tool | `openshell sandbox create -- claude` | | Create with custom policy | `openshell sandbox create --policy ./p.yaml` | | Connect to sandbox | `openshell sandbox connect ` | | Stream live logs | `openshell logs --tail` | +| Incremental policy update | `openshell policy update --add-endpoint host:443:read-only:rest:enforce --binary /usr/bin/curl --wait` | | Pull current policy | `openshell policy get --full > p.yaml` | | Push updated policy | `openshell policy set --policy p.yaml --wait` | | Policy revision history | `openshell policy list ` | @@ -555,7 +549,7 @@ $ openshell sandbox upload --help | Configure gateway inference | `openshell inference set --provider P --model M` | | View gateway inference | `openshell inference get` | | Delete sandbox | `openshell sandbox delete ` | -| Destroy cluster | `openshell gateway destroy` | +| Remove gateway registration | `openshell gateway destroy --name ` | | Self-teach any command | `openshell --help` | ## Companion Skills @@ -563,6 +557,6 @@ $ openshell sandbox upload --help | Skill | When to use | |-------|------------| | `generate-sandbox-policy` | Creating or modifying policy YAML content (network rules, L7 inspection, access presets, endpoint configuration) | -| `debug-openshell-cluster` | Diagnosing cluster startup or health failures | +| `debug-openshell-cluster` | Diagnosing gateway deployment, runtime, or health failures | | `debug-inference` | Diagnosing `inference.local`, host-backed local inference, and provider base URL issues | | `tui-development` | Developing features for the OpenShell TUI (`openshell term`) | diff --git a/.agents/skills/openshell-cli/cli-reference.md b/.agents/skills/openshell-cli/cli-reference.md index e344f20dfd..3256aef423 100644 --- a/.agents/skills/openshell-cli/cli-reference.md +++ b/.agents/skills/openshell-cli/cli-reference.md @@ -25,8 +25,8 @@ Quick-reference for the `openshell` command-line interface. For workflow guidanc ``` openshell ├── gateway -│ ├── start [opts] -│ ├── stop [opts] +│ ├── add [opts] +│ ├── login [name] │ ├── destroy [opts] │ ├── info [--name] │ └── select [name] @@ -73,40 +73,37 @@ openshell ## Gateway Commands -### `openshell gateway start` +### `openshell gateway add ` -Provision or start a cluster (local or remote). - -| Flag | Default | Description | -|------|---------|-------------| -| `--name ` | `openshell` | Cluster name | -| `--remote ` | none | SSH destination for remote deployment | -| `--ssh-key ` | none | SSH private key for remote deployment | -| `--port ` | 8080 | Host port mapped to gateway | -| `--gateway-host ` | none | Override gateway host in metadata | -| `--recreate` | false | Destroy and recreate from scratch if a gateway already exists (skips interactive prompt) | - -### `openshell gateway stop` - -Stop a cluster (preserves state for later restart). +Register an existing gateway endpoint. | Flag | Description | |------|-------------| -| `--name ` | Cluster name (defaults to active) | -| `--remote ` | SSH destination | -| `--ssh-key ` | SSH private key | +| `--name ` | Gateway name | +| `--local` | Register a local endpoint, commonly a trusted port-forward | +| `--remote ` | Register a remote gateway associated with an SSH destination | +| `--ssh-key ` | SSH private key for the remote host | + +Examples: + +- `openshell gateway add http://127.0.0.1:8080 --local --name local` +- `openshell gateway add https://gateway.example.com --name production` ### `openshell gateway destroy` -Destroy a cluster and all its state. Same flags as `stop`. +Remove a gateway registration. For Helm deployments this affects local CLI metadata only; it does not uninstall the Helm release. + +### `openshell gateway login [name]` + +Refresh browser-based authentication for a gateway behind an edge proxy. ### `openshell gateway info` -Show deployment details: endpoint and remote host. +Show gateway details: endpoint, auth mode, and remote host metadata when present. | Flag | Description | |------|-------------| -| `--name ` | Cluster name (defaults to active) | +| `--name ` | Gateway name (defaults to active) | ### `openshell gateway select [name]` @@ -118,7 +115,7 @@ Set the active gateway. Writes to `~/.config/openshell/active_gateway`. When cal ### `openshell doctor logs` -Fetch logs from the gateway Docker container. +Fetch logs when gateway metadata supports it. For Helm deployments, prefer `kubectl -n openshell logs statefulset/openshell`. | Flag | Default | Description | |------|---------|-------------| @@ -130,8 +127,7 @@ Fetch logs from the gateway Docker container. ### `openshell doctor exec -- ` -Run a command inside the gateway container with KUBECONFIG pre-configured. -Launches an interactive `docker exec` session (tunnelled over SSH for remote gateways). +Run a diagnostic command when gateway metadata supports it. For Helm deployments, prefer direct `kubectl` and `helm` commands. | Flag | Default | Description | |------|---------|-------------| @@ -140,9 +136,9 @@ Launches an interactive `docker exec` session (tunnelled over SSH for remote gat | `--ssh-key ` | none | SSH private key for remote gateways | Examples: -- `openshell doctor exec -- kubectl get pods -A` -- `openshell doctor exec -- k9s` -- `openshell doctor exec -- sh` (interactive shell) +- `kubectl -n openshell get pods` +- `kubectl -n openshell logs statefulset/openshell` +- `helm -n openshell status openshell` --- @@ -158,7 +154,7 @@ Show server connectivity and version for the active gateway. ### `openshell sandbox create [OPTIONS] [-- COMMAND...]` -Create a sandbox, wait for readiness, then connect or execute the trailing command. Auto-bootstraps a cluster if none exists. +Create a sandbox through the active gateway, wait for readiness, then connect or execute the trailing command. | Flag | Description | |------|-------------| @@ -169,19 +165,19 @@ Create a sandbox, wait for readiness, then connect or execute the trailing comma | `--provider ` | Provider to attach (repeatable) | | `--policy ` | Path to custom policy YAML | | `--forward ` | Forward local port to sandbox (keeps the sandbox alive) | -| `--remote ` | SSH destination for auto-bootstrap | -| `--ssh-key ` | SSH private key for auto-bootstrap | | `--tty` | Force pseudo-terminal allocation | | `--no-tty` | Disable pseudo-terminal allocation | -| `--bootstrap` | Auto-bootstrap a gateway if none is available (skips interactive prompt) | -| `--no-bootstrap` | Never auto-bootstrap; error immediately if no gateway is available | | `--auto-providers` | Auto-create missing providers from local credentials (skips interactive prompt) | | `--no-auto-providers` | Never auto-create providers; skip missing providers silently | | `[-- COMMAND...]` | Command to execute (defaults to interactive shell) | ### `openshell sandbox get ` -Show sandbox details (id, name, namespace, phase, policy). +Show sandbox details (id, name, namespace, phase) and the **active** policy from the gateway (same source whether policy is sandbox-scoped or global). Metadata includes **Policy source** (`sandbox` or `global`) and **Revision** (global policy row when source is global, otherwise sandbox policy row). + +| Flag | Description | +|------|-------------| +| `--policy-only` | Print only the active policy YAML to stdout (same policy as above; use for scripts and piping) | ### `openshell sandbox list` @@ -268,9 +264,32 @@ View sandbox logs. Supports one-shot and streaming. ## Policy Commands +### `openshell policy update ` + +Incrementally merge live network policy changes into the current sandbox policy. Multiple flags in one invocation are applied as one atomic batch and create at most one new revision. + +| Flag | Default | Description | +|------|---------|-------------| +| `--add-endpoint ` | repeatable | `host:port[:access[:protocol[:enforcement]]]`. Adds or merges an endpoint. `access`: `read-only`, `read-write`, `full`. `protocol`: `rest`, `sql`. `enforcement`: `enforce`, `audit`. | +| `--remove-endpoint ` | repeatable | `host:port`. Removes the endpoint or just the requested port from a multi-port endpoint. | +| `--add-allow ` | repeatable | `host:port:METHOD:path_glob`. Adds REST allow rules to an existing `protocol: rest` endpoint. | +| `--add-deny ` | repeatable | `host:port:METHOD:path_glob`. Adds REST deny rules to an existing `protocol: rest` endpoint that already has an allow base. | +| `--remove-rule ` | repeatable | Deletes a named network rule. | +| `--binary ` | repeatable | Adds binaries to each `--add-endpoint` rule. Valid only with `--add-endpoint`. | +| `--rule-name ` | none | Overrides the generated rule name. Valid only when exactly one `--add-endpoint` is provided. | +| `--dry-run` | false | Preview the merged policy locally without sending an update to the gateway. | +| `--wait` | false | Wait for the sandbox to confirm the new policy revision is loaded. | +| `--timeout ` | 60 | Timeout for `--wait`. | + +Notes: + +- `--add-allow` and `--add-deny` currently operate only on `protocol: rest` endpoints. +- `--wait` cannot be combined with `--dry-run`. +- Use `policy set` when replacing the full policy or changing static sections. + ### `openshell policy set --policy ` -Update the policy on a live sandbox. Only the dynamic `network_policies` field can be changed at runtime. +Replace the full policy on a live sandbox. Only the dynamic `network_policies` field can be changed at runtime. | Flag | Default | Description | |------|---------|-------------| diff --git a/.agents/skills/triage-issue/SKILL.md b/.agents/skills/triage-issue/SKILL.md index e4998a0569..54e726a5d9 100644 --- a/.agents/skills/triage-issue/SKILL.md +++ b/.agents/skills/triage-issue/SKILL.md @@ -85,19 +85,7 @@ Check whether the issue body contains a substantive agent diagnostic section. Lo ```bash gh issue edit --add-label "state:triage-needed" ``` -2. Post a comment with the triage marker: - ``` - > **📋 triage-agent** - > - > This issue was opened without an agent investigation. - > - > OpenShell is an agent-first project - before we triage this, please point your coding agent at the repo and have it investigate. Your agent can load skills like `debug-openshell-cluster` (for cluster issues), `debug-inference` (for inference setup issues), `openshell-cli` (for usage questions), or `generate-sandbox-policy` (for policy help). - > - > See [CONTRIBUTING.md](https://github.com/NVIDIA/OpenShell/blob/main/CONTRIBUTING.md#before-you-open-an-issue) for the full workflow. - > - > **Classification:** needs-more-info (agent diagnostic required) - ``` -3. Stop. Do not proceed with diagnosis until the reporter provides diagnostics. +2. Do not post a standalone redirect comment. Report the missing diagnostic to the operator and stop unless a human explicitly asks you to continue triage anyway. **If the diagnostic section is substantive**, proceed to Step 4. @@ -122,7 +110,7 @@ Based on the sub-agent's analysis, also attempt to validate the report directly: - For bug reports: check the relevant code paths, look for the described failure mode - For feature requests: assess feasibility against the existing architecture -- For cluster/infrastructure issues: reference the `debug-openshell-cluster` skill's known failure patterns +- For gateway deployment or infrastructure issues: reference the `debug-openshell-cluster` skill's known failure patterns - For inference and provider-topology issues: reference the `debug-inference` skill's known failure patterns - For CLI/usage issues: reference the `openshell-cli` skill's command reference diff --git a/.agents/skills/tui-development/SKILL.md b/.agents/skills/tui-development/SKILL.md index 80984759c5..bbd9f1ecd4 100644 --- a/.agents/skills/tui-development/SKILL.md +++ b/.agents/skills/tui-development/SKILL.md @@ -19,7 +19,7 @@ The OpenShell TUI is a ratatui-based terminal UI for the OpenShell platform. It - `tonic` with TLS — gRPC client for the OpenShell gateway - `tokio` — async runtime for event loop, spawned tasks, and mpsc channels - `openshell-core` — proto-generated types (`OpenShellClient`, request/response structs) - - `openshell-bootstrap` — cluster discovery (`list_clusters()`) + - `openshell-bootstrap` — gateway discovery (`list_gateways()`) - **Theme:** Adaptive dark/light via `Theme` struct — NVIDIA-branded green accents. Controlled by `--theme` flag, `OPENSHELL_THEME` env var, or auto-detection. ## 2. Domain Object Hierarchy @@ -33,7 +33,7 @@ Gateway (discovered via openshell_bootstrap::list_gateways()) ``` - **Gateways** are discovered from on-disk config via `openshell_bootstrap::list_gateways()`. Each gateway has a name, endpoint, and local/remote flag. -- **Sandboxes** belong to the active cluster. Fetched via `ListSandboxes` gRPC call with a periodic tick refresh. Each sandbox has: `id`, `name`, `phase`, `created_at_ms`, and `spec.template.image`. +- **Sandboxes** belong to the active gateway. Fetched via `ListSandboxes` gRPC call with a periodic tick refresh. Each sandbox has: `id`, `name`, `phase`, `created_at_ms`, and `spec.template.image`. - **Logs** belong to a single sandbox. Initial batch fetched via `GetSandboxLogs` (500 lines), then live-tailed via `WatchSandbox` with `follow_logs: true`. The **title bar** always reflects this hierarchy, reading left-to-right from general to specific: @@ -90,7 +90,7 @@ Every frame renders four vertical regions: ``` ┌─────────────────────────────────────────────┐ -│ Title bar (1 row) — brand + cluster + context│ +│ Title bar (1 row) — brand + gateway + context│ ├─────────────────────────────────────────────┤ │ │ │ Main content (flexible) │ @@ -270,7 +270,7 @@ TUI actions should parallel `openshell` CLI commands so users have familiar ment | `openshell sandbox list` | Sandbox table on Dashboard | | `openshell sandbox delete ` | `[d]` on sandbox detail, then `[y]` to confirm | | `openshell logs ` | `[l]` on sandbox detail to open log viewer | -| `openshell status` | Status in title bar + cluster list | +| `openshell status` | Status in title bar + gateway list | When adding new TUI features, check what the CLI offers and maintain consistency. @@ -364,7 +364,7 @@ lib.rs (event loop, gRPC, async tasks) ├── theme.rs (colors + styles) └── ui/ ├── mod.rs (draw dispatcher, chrome) - ├── dashboard.rs (cluster list + sandbox table layout) + ├── dashboard.rs (gateway list + sandbox table layout) ├── sandboxes.rs (sandbox table widget) ├── sandbox_detail.rs (detail view) └── sandbox_logs.rs (log viewer) @@ -415,7 +415,7 @@ All gRPC calls use a 5-second timeout: tokio::time::timeout(Duration::from_secs(5), client.health(req)).await ``` -The connect timeout for cluster switching is 10 seconds with HTTP/2 keepalive at 10-second intervals. +The connect timeout for gateway switching is 10 seconds with HTTP/2 keepalive at 10-second intervals. ### Log streaming lifecycle @@ -443,7 +443,7 @@ The connect timeout for cluster switching is 10 seconds with HTTP/2 keepalive at # Build the crate cargo build -p openshell-tui -# Run the TUI against the active cluster +# Run the TUI against the active gateway mise run term # Run with cargo-watch for hot-reload during development @@ -466,13 +466,21 @@ mise run pre-commit ### Gateway changes -If you change sandbox or server code that affects the backend, redeploy the gateway: +If you change sandbox or server code that affects the backend, restart or redeploy the gateway for the compute platform you are using. + +For Docker-backed local development: + +```bash +mise run gateway:docker +``` + +For Kubernetes Helm deployments: ```bash -mise run cluster:deploy all +helm upgrade --install openshell deploy/helm/openshell --namespace openshell ``` -To pick up new sandbox images after changing sandbox code, delete the pod manually so it gets recreated: +For Kubernetes, pick up new sandbox images after changing sandbox code by deleting the pod manually so it gets recreated: ```bash kubectl delete pod -n diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index b73df3d205..b569db2a7c 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,9 +1,5 @@ # Broad ownership — core team reviews everything -* @NVIDIA/openshell-codeowners - -# Agent infrastructure — tighter review -.agents/ @NVIDIA/openshell-codeowners -AGENTS.md @NVIDIA/openshell-codeowners +* @NVIDIA/openshell-codeowners @mrunalp @maxamillion @derekwaynecarr # Vouch list — maintainers only (bot commits bypass, but manual edits need review) .github/VOUCHED.td @NVIDIA/openshell-codeowners diff --git a/.github/actions/pr-gate/action.yml b/.github/actions/pr-gate/action.yml new file mode 100644 index 0000000000..5d4b9c1833 --- /dev/null +++ b/.github/actions/pr-gate/action.yml @@ -0,0 +1,62 @@ +name: PR Gate +description: > + Resolve PR metadata for a `pull-request/` push from copy-pr-bot and decide + whether the workflow should run. Sets `should-run=true` only when the pushed + SHA still matches the PR head SHA. If `required_label` is provided, the PR + must also carry that label. For non-`push` events (e.g. `workflow_dispatch`), + always sets `should-run=true`. + +inputs: + required_label: + description: Optional PR label required to enable the run (e.g. "test:e2e"). + required: false + default: "" + +outputs: + should_run: + description: "true if the workflow should proceed, false otherwise" + value: ${{ steps.gate.outputs.should_run }} + +runs: + using: composite + steps: + - id: get_pr_info + if: github.event_name == 'push' + continue-on-error: true + uses: nv-gha-runners/get-pr-info@main + + - id: gate + shell: bash + env: + EVENT_NAME: ${{ github.event_name }} + GITHUB_SHA_VALUE: ${{ github.sha }} + GET_PR_INFO_OUTCOME: ${{ steps.get_pr_info.outcome }} + PR_INFO: ${{ steps.get_pr_info.outputs.pr-info }} + REQUIRED_LABEL: ${{ inputs.required_label }} + run: | + if [ "$EVENT_NAME" != "push" ]; then + echo "should_run=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + + if [ "$GET_PR_INFO_OUTCOME" != "success" ]; then + echo "should_run=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + + head_sha="$(jq -r '.head.sha' <<< "$PR_INFO")" + if [ -z "$REQUIRED_LABEL" ]; then + has_label=true + else + has_label="$(jq -r --arg L "$REQUIRED_LABEL" '[.labels[].name] | index($L) != null' <<< "$PR_INFO")" + fi + + # Only trust copied pull-request/* pushes that still match the PR head + # SHA and, when configured, carry the required label. + if [ "$head_sha" = "$GITHUB_SHA_VALUE" ] && [ "$has_label" = "true" ]; then + should_run=true + else + should_run=false + fi + + echo "should_run=$should_run" >> "$GITHUB_OUTPUT" diff --git a/.github/actions/setup-buildx/action.yml b/.github/actions/setup-buildx/action.yml index f7c8dbbd01..41210035fb 100644 --- a/.github/actions/setup-buildx/action.yml +++ b/.github/actions/setup-buildx/action.yml @@ -1,24 +1,41 @@ name: Setup Docker Buildx description: > - Create a multi-arch Docker Buildx builder using remote BuildKit nodes. - The builder is automatically removed when the job finishes (cleanup is - enabled by default in docker/setup-buildx-action). + Create a Docker Buildx builder. Two modes: + * driver=remote (default) — multi-arch builder against in-cluster BuildKit + pods. Requires EKS connectivity. Behaviour unchanged from prior versions. + * driver=local — single-node buildx on the local docker-container driver. + Pair with cache-to/cache-from=type=gha on build steps for persistence. + Works on nv-gha-runners; no EKS needed. + Cleanup is automatic when the job finishes (docker/setup-buildx-action default). inputs: + driver: + description: "buildx driver: 'remote' or 'local'" + default: remote amd64-endpoint: - description: BuildKit endpoint for linux/amd64 + description: BuildKit endpoint for linux/amd64 (remote driver only) default: tcp://buildkit-amd64.buildkit:1234 arm64-endpoint: - description: BuildKit endpoint for linux/arm64 + description: BuildKit endpoint for linux/arm64 (remote driver only) default: tcp://buildkit-arm64.buildkit:1234 name: description: Builder instance name default: openshell + buildkitd-config: + description: > + Path to a buildkitd.toml to configure the builder with (e.g. the + nv-gha-runners Docker Hub mirror at /etc/buildkit/buildkitd.toml). + Must be readable *from where this action runs* — in a containerized + job that means the caller must bind-mount the host path into the job + container (e.g. `volumes: [/etc/buildkit:/etc/buildkit:ro]`). Empty + disables the config (default). + default: "" runs: using: composite steps: - - name: Set up Docker Buildx + - name: Set up Docker Buildx (remote) + if: inputs.driver == 'remote' uses: docker/setup-buildx-action@v3 with: name: ${{ inputs.name }} @@ -28,3 +45,13 @@ runs: append: | - endpoint: ${{ inputs.arm64-endpoint }} platforms: linux/arm64 + buildkitd-config: ${{ inputs.buildkitd-config }} + + - name: Set up Docker Buildx (local) + if: inputs.driver == 'local' + uses: docker/setup-buildx-action@v3 + with: + name: ${{ inputs.name }} + driver: docker-container + platforms: linux/amd64,linux/arm64 + buildkitd-config: ${{ inputs.buildkitd-config }} diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000000..29aeab9aa1 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,8 @@ +version: 2 +updates: + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "daily" + cooldown: + default-days: 2 diff --git a/.github/workflows/branch-checks.yml b/.github/workflows/branch-checks.yml index 3d31f6e9f6..115e0b5c17 100644 --- a/.github/workflows/branch-checks.yml +++ b/.github/workflows/branch-checks.yml @@ -1,56 +1,107 @@ name: Branch Checks on: - pull_request: + push: + branches: + - "pull-request/[0-9]+" + workflow_dispatch: env: CARGO_TERM_COLOR: always CARGO_INCREMENTAL: "0" MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + SCCACHE_GHA_ENABLED: "true" permissions: contents: read packages: read +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: + pr_metadata: + name: Resolve PR metadata + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read + outputs: + should_run: ${{ steps.gate.outputs.should_run }} + steps: + - uses: actions/checkout@v6 + + - id: gate + uses: ./.github/actions/pr-gate + + mise-lockfile: + name: mise Lockfile + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' + runs-on: linux-amd64-cpu8 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + + - name: Mark workspace as safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Verify mise.lock is in sync with mise.toml + run: | + mise lock + if ! git diff --exit-code mise.lock; then + echo "::error::mise.lock is out of sync with mise.toml. Run 'mise lock' locally and commit the result." >&2 + exit 1 + fi + license-headers: name: License Headers - runs-on: build-amd64 + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' + runs-on: linux-amd64-cpu8 container: image: ghcr.io/nvidia/openshell/ci:latest credentials: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Install tools - run: mise install + run: mise install --locked - name: Check license headers run: mise run license:check rust: name: Rust (${{ matrix.runner }}) + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' strategy: fail-fast: false matrix: - runner: [build-amd64, build-arm64] + runner: [linux-amd64-cpu8, linux-arm64-cpu8] runs-on: ${{ matrix.runner }} + env: + SCCACHE_GHA_VERSION: branch-checks-rust-${{ matrix.runner }} container: image: ghcr.io/nvidia/openshell/ci:latest credentials: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Install tools - run: mise install + run: mise install --locked - - name: Configure sccache remote cache - if: vars.SCCACHE_MEMCACHED_ENDPOINT != '' - run: echo "SCCACHE_MEMCACHED_ENDPOINT=${{ vars.SCCACHE_MEMCACHED_ENDPOINT }}" >> "$GITHUB_ENV" + - name: Configure GHA sccache backend + uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9 - name: Cache Rust target and registry uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 @@ -72,14 +123,24 @@ jobs: - name: sccache stats if: always() - run: mise x -- sccache --show-stats + run: | + set +e + stats_bin="${SCCACHE_PATH:-sccache}" + "$stats_bin" --show-stats + status=$? + if [[ $status -ne 0 ]]; then + echo "::warning::sccache stats unavailable (exit $status)" + fi + exit 0 python: name: Python (${{ matrix.runner }}) + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' strategy: fail-fast: false matrix: - runner: [build-amd64, build-arm64] + runner: [linux-amd64-cpu8, linux-arm64-cpu8] runs-on: ${{ matrix.runner }} container: image: ghcr.io/nvidia/openshell/ci:latest @@ -87,10 +148,10 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Install tools - run: mise install + run: mise install --locked - name: Install dependencies run: uv sync --frozen @@ -100,3 +161,22 @@ jobs: - name: Test run: mise run test:python + + markdown: + name: Markdown + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' + runs-on: linux-amd64-cpu8 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + + - name: Install tools + run: mise install --locked + + - name: Lint + run: mise run markdown:lint diff --git a/.github/workflows/branch-docs.yml b/.github/workflows/branch-docs.yml index 762d2d1fbb..1368bc775b 100644 --- a/.github/workflows/branch-docs.yml +++ b/.github/workflows/branch-docs.yml @@ -19,7 +19,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Check Fern preview availability id: fern-preview diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index ad53bb6358..cb2f5440e3 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -1,33 +1,47 @@ name: Branch E2E Checks on: - pull_request: - types: [opened, synchronize, reopened, labeled] + push: + branches: + - "pull-request/[0-9]+" + workflow_dispatch: {} -permissions: - contents: read - packages: write +permissions: {} jobs: + pr_metadata: + name: Resolve PR metadata + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read + outputs: + should_run: ${{ steps.gate.outputs.should_run }} + steps: + - uses: actions/checkout@v6 + - id: gate + uses: ./.github/actions/pr-gate + with: + required_label: test:e2e + build-gateway: - if: contains(github.event.pull_request.labels.*.name, 'test:e2e') + needs: [pr_metadata] + if: needs.pr_metadata.outputs.should_run == 'true' + permissions: + contents: read + packages: write uses: ./.github/workflows/docker-build.yml with: component: gateway platform: linux/arm64 - runner: build-arm64 - - build-cluster: - if: contains(github.event.pull_request.labels.*.name, 'test:e2e') - uses: ./.github/workflows/docker-build.yml - with: - component: cluster - platform: linux/arm64 - runner: build-arm64 e2e: - needs: [build-gateway, build-cluster] + needs: [pr_metadata, build-gateway] + if: needs.pr_metadata.outputs.should_run == 'true' + permissions: + contents: read + packages: read uses: ./.github/workflows/e2e-test.yml with: image-tag: ${{ github.sha }} - runner: build-arm64 + runner: linux-arm64-cpu8 diff --git a/.github/workflows/ci-image.yml b/.github/workflows/ci-image.yml index 2f7737d961..8b05091697 100644 --- a/.github/workflows/ci-image.yml +++ b/.github/workflows/ci-image.yml @@ -6,6 +6,7 @@ on: paths: - 'deploy/docker/Dockerfile.ci' - 'mise.toml' + - 'mise.lock' - 'tasks/**' - '.github/workflows/ci-image.yml' workflow_dispatch: @@ -20,10 +21,21 @@ permissions: jobs: build-ci-image: - name: Build - runs-on: build-amd64 + name: Build (${{ matrix.arch }}) + strategy: + fail-fast: false + matrix: + include: + - arch: amd64 + platform: linux/amd64 + runner: linux-amd64-cpu8 + - arch: arm64 + platform: linux/arm64 + runner: linux-arm64-cpu8 + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Log in to GitHub Container Registry uses: docker/login-action@v3 @@ -32,18 +44,66 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} + - name: Resolve BuildKit config + id: buildkit + run: | + if [[ -r /etc/buildkit/buildkitd.toml ]]; then + echo "config=/etc/buildkit/buildkitd.toml" >> "$GITHUB_OUTPUT" + else + echo "config=" >> "$GITHUB_OUTPUT" + fi + - name: Set up Docker Buildx uses: ./.github/actions/setup-buildx + with: + driver: local + buildkitd-config: ${{ steps.buildkit.outputs.config }} - name: Build and push CI image env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + ARCH_IMAGE: ${{ env.CI_IMAGE }}:${{ github.sha }}-${{ matrix.arch }} run: | + set -euo pipefail docker buildx build \ - --platform linux/amd64,linux/arm64 \ + --builder openshell \ + --platform "${{ matrix.platform }}" \ --secret id=MISE_GITHUB_TOKEN,env=MISE_GITHUB_TOKEN \ + --cache-from "type=gha,scope=ci-image-${{ matrix.arch }}" \ + --cache-to "type=gha,mode=max,scope=ci-image-${{ matrix.arch }}" \ --push \ - -t ${{ env.CI_IMAGE }}:${{ github.sha }} \ - -t ${{ env.CI_IMAGE }}:latest \ + -t "$ARCH_IMAGE" \ -f deploy/docker/Dockerfile.ci \ . + + - name: Smoke check CI image + env: + ARCH_IMAGE: ${{ env.CI_IMAGE }}:${{ github.sha }}-${{ matrix.arch }} + run: | + set -euo pipefail + docker run --rm --platform "${{ matrix.platform }}" "$ARCH_IMAGE" mise --version + docker run --rm --platform "${{ matrix.platform }}" "$ARCH_IMAGE" gh --version + docker run --rm --platform "${{ matrix.platform }}" "$ARCH_IMAGE" docker buildx version + + merge-ci-image: + name: Merge manifest + needs: build-ci-image + runs-on: linux-amd64-cpu8 + timeout-minutes: 10 + steps: + - name: Log in to GitHub Container Registry + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Create multi-arch manifests + run: | + set -euo pipefail + docker buildx imagetools create \ + --prefer-index=false \ + -t "${CI_IMAGE}:${GITHUB_SHA}" \ + -t "${CI_IMAGE}:latest" \ + "${CI_IMAGE}:${GITHUB_SHA}-amd64" \ + "${CI_IMAGE}:${GITHUB_SHA}-arm64" diff --git a/.github/workflows/dco.yml b/.github/workflows/dco.yml index 88660b1e54..53339f4bf9 100644 --- a/.github/workflows/dco.yml +++ b/.github/workflows/dco.yml @@ -38,7 +38,7 @@ jobs: path-to-signatures: "dco-signatures.json" path-to-document: "https://github.com/NVIDIA/OpenShell/blob/main/DCO" branch: "signatures" - allowlist: dependabot + allowlist: "dependabot[bot]" create-file-commit-message: "chore: create file to store dco signatures" signed-commit-message: "chore: $contributorName has signed the dco in #$pullRequestNo" custom-notsigned-prcomment: >- diff --git a/.github/workflows/deb-package.yml b/.github/workflows/deb-package.yml new file mode 100644 index 0000000000..2877487582 --- /dev/null +++ b/.github/workflows/deb-package.yml @@ -0,0 +1,92 @@ +name: Debian Package + +on: + workflow_call: + inputs: + deb-version: + required: true + type: string + checkout-ref: + required: true + type: string + +permissions: + contents: read + packages: read + +defaults: + run: + shell: bash + +jobs: + build-deb-linux: + name: Build Debian Package (Linux ${{ matrix.arch }}) + strategy: + matrix: + include: + - arch: amd64 + runner: linux-amd64-cpu8 + deb_arch: amd64 + cli_target: x86_64-unknown-linux-musl + gnu_target: x86_64-unknown-linux-gnu + - arch: arm64 + runner: linux-arm64-cpu8 + deb_arch: arm64 + cli_target: aarch64-unknown-linux-musl + gnu_target: aarch64-unknown-linux-gnu + runs-on: ${{ matrix.runner }} + timeout-minutes: 20 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs['checkout-ref'] }} + + - name: Download CLI artifact + uses: actions/download-artifact@v4 + with: + name: cli-linux-${{ matrix.arch }} + path: package-input/ + + - name: Download gateway artifact + uses: actions/download-artifact@v4 + with: + name: gateway-binary-linux-${{ matrix.arch }} + path: package-input/ + + - name: Download VM driver artifact + uses: actions/download-artifact@v4 + with: + name: driver-vm-linux-${{ matrix.arch }} + path: package-input/ + + - name: Extract package inputs + run: | + set -euo pipefail + mkdir -p package-binaries + tar -xzf "package-input/openshell-${{ matrix.cli_target }}.tar.gz" -C package-binaries + tar -xzf "package-input/openshell-gateway-${{ matrix.gnu_target }}.tar.gz" -C package-binaries + tar -xzf "package-input/openshell-driver-vm-${{ matrix.gnu_target }}.tar.gz" -C package-binaries + ls -lah package-binaries + + - name: Build Debian package + run: | + set -euo pipefail + OPENSHELL_CLI_BINARY="${PWD}/package-binaries/openshell" \ + OPENSHELL_GATEWAY_BINARY="${PWD}/package-binaries/openshell-gateway" \ + OPENSHELL_DRIVER_VM_BINARY="${PWD}/package-binaries/openshell-driver-vm" \ + OPENSHELL_DEB_VERSION="${{ inputs['deb-version'] }}" \ + OPENSHELL_DEB_ARCH="${{ matrix.deb_arch }}" \ + OPENSHELL_OUTPUT_DIR=artifacts \ + tasks/scripts/package-deb.sh + + - name: Upload Debian package artifact + uses: actions/upload-artifact@v4 + with: + name: deb-linux-${{ matrix.arch }} + path: artifacts/*.deb + retention-days: 5 diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index 16a8447c99..3b3aa1cb82 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -4,11 +4,11 @@ on: workflow_call: inputs: component: - description: "Component to build (gateway, cluster)" + description: "Component to build (gateway, supervisor, cluster)" required: true type: string timeout-minutes: - description: "Job timeout in minutes" + description: "Per-arch Docker image job timeout in minutes" required: false type: number default: 20 @@ -23,29 +23,145 @@ on: type: string default: "linux/amd64,linux/arm64" runner: - description: "GitHub Actions runner label" + description: "Deprecated; per-arch native runners are selected automatically" required: false type: string - default: "build-amd64" + default: "linux-amd64-cpu8" cargo-version: description: "Pre-computed cargo version (skips internal git-based computation)" required: false type: string default: "" + image-tag: + description: "Image tag base to build/push (defaults to the GitHub SHA)" + required: false + type: string + default: "" env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} permissions: contents: read packages: write +defaults: + run: + shell: bash + jobs: + resolve: + name: Resolve build plan + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.resolve.outputs.matrix }} + platform_count: ${{ steps.resolve.outputs.platform_count }} + arches: ${{ steps.resolve.outputs.arches }} + binary_component: ${{ steps.resolve.outputs.binary_component }} + binary_name: ${{ steps.resolve.outputs.binary_name }} + artifact_prefix: ${{ steps.resolve.outputs.artifact_prefix }} + image_tag_base: ${{ steps.resolve.outputs.image_tag_base }} + steps: + - name: Resolve component and platform matrix + id: resolve + run: | + set -euo pipefail + + component="${{ inputs.component }}" + case "$component" in + gateway) + binary_component=gateway + binary_name=openshell-gateway + ;; + supervisor|cluster) + binary_component=sandbox + binary_name=openshell-sandbox + ;; + *) + echo "unsupported component: $component" >&2 + exit 1 + ;; + esac + + image_tag_base="${{ inputs['image-tag'] }}" + if [[ -z "$image_tag_base" ]]; then + image_tag_base="$GITHUB_SHA" + fi + + platform_input="${{ inputs.platform }}" + platform_input="${platform_input//[[:space:]]/}" + if [[ -z "$platform_input" ]]; then + echo "platform input must not be empty" >&2 + exit 1 + fi + + IFS=',' read -r -a platforms <<< "$platform_input" + matrix='{"include":[' + arches=() + count=0 + + for platform in "${platforms[@]}"; do + case "$platform" in + linux/amd64) + arch=amd64 + runner=linux-amd64-cpu8 + ;; + linux/arm64) + arch=arm64 + runner=linux-arm64-cpu8 + ;; + *) + echo "unsupported platform: $platform" >&2 + echo "supported platforms: linux/amd64, linux/arm64" >&2 + exit 1 + ;; + esac + + if [[ $count -gt 0 ]]; then + matrix+=',' + fi + matrix+='{"platform":"'"$platform"'","arch":"'"$arch"'","runner":"'"$runner"'"}' + arches+=("$arch") + count=$((count + 1)) + done + + matrix+=']}' + { + echo "matrix=$matrix" + echo "platform_count=$count" + echo "arches=${arches[*]}" + echo "binary_component=$binary_component" + echo "binary_name=$binary_name" + echo "artifact_prefix=rust-binary-${component}-${binary_component}" + echo "image_tag_base=$image_tag_base" + } >> "$GITHUB_OUTPUT" + + rust-binary: + name: Rust ${{ needs.resolve.outputs.binary_component }} (${{ matrix.arch }}) + needs: resolve + permissions: + contents: read + packages: read + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.resolve.outputs.matrix) }} + uses: ./.github/workflows/shadow-rust-native-build.yml + with: + component: ${{ needs.resolve.outputs.binary_component }} + arch: ${{ matrix.arch }} + cargo-version: ${{ inputs['cargo-version'] }} + features: openshell-core/dev-settings + artifact-name: ${{ needs.resolve.outputs.artifact_prefix }}-linux-${{ matrix.arch }} + secrets: inherit + build: - name: Build ${{ inputs.component }} - runs-on: ${{ inputs.runner }} - timeout-minutes: ${{ inputs.timeout-minutes }} + name: Build ${{ inputs.component }} (${{ matrix.arch }}) + needs: [resolve, rust-binary] + runs-on: ${{ matrix.runner }} + timeout-minutes: ${{ inputs['timeout-minutes'] }} + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.resolve.outputs.matrix) }} container: image: ghcr.io/nvidia/openshell/ci:latest credentials: @@ -54,43 +170,115 @@ jobs: options: --privileged volumes: - /var/run/docker.sock:/var/run/docker.sock + # Expose the nv-gha-runners buildkitd.toml registry mirror config + # inside the container so setup-buildx can read it. + - /etc/buildkit:/etc/buildkit:ro env: - IMAGE_TAG: ${{ github.sha }} + IMAGE_TAG: ${{ needs.resolve.outputs.platform_count == '1' && needs.resolve.outputs.image_tag_base || format('{0}-{1}', needs.resolve.outputs.image_tag_base, matrix.arch) }} IMAGE_REGISTRY: ghcr.io/nvidia/openshell DOCKER_PUSH: ${{ inputs.push && '1' || '0' }} - DOCKER_PLATFORM: ${{ inputs.platform }} + DOCKER_PLATFORM: ${{ matrix.platform }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 - name: Mark workspace safe for git run: git config --global --add safe.directory "$GITHUB_WORKSPACE" - - name: Fetch tags - run: git fetch --tags --force - - - name: Compute cargo version - id: version - run: | - set -eu - if [[ -n "${{ inputs.cargo-version }}" ]]; then - echo "cargo_version=${{ inputs.cargo-version }}" >> "$GITHUB_OUTPUT" - else - echo "cargo_version=$(uv run python tasks/scripts/release.py get-version --cargo)" >> "$GITHUB_OUTPUT" - fi + - name: Install tools + run: mise install --locked - name: Log in to GHCR + if: ${{ inputs.push }} run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin - - name: Set up Docker Buildx + - name: Set up buildx (local driver) uses: ./.github/actions/setup-buildx + with: + driver: local + buildkitd-config: /etc/buildkit/buildkitd.toml + + - name: Download Rust binary artifact + uses: actions/download-artifact@v4 + with: + name: ${{ needs.resolve.outputs.artifact_prefix }}-linux-${{ matrix.arch }} + path: prebuilt-rust-binary + + - name: Stage Rust binary in Docker build context + run: | + set -euo pipefail + binary="${{ needs.resolve.outputs.binary_name }}" + download_dir="prebuilt-rust-binary" + stage="deploy/docker/.build/prebuilt-binaries/${{ matrix.arch }}" + found="$(find "$download_dir" -type f -name "$binary" -print -quit)" + if [[ -z "$found" ]]; then + echo "missing downloaded artifact file: $binary" >&2 + find "$download_dir" -maxdepth 4 -type f -print >&2 || true + exit 1 + fi + mkdir -p "$stage" + install -m 0755 "$found" "$stage/$binary" + ls -lh "$stage/" - name: Build ${{ inputs.component }} image env: DOCKER_BUILDER: openshell - OPENSHELL_CARGO_VERSION: ${{ steps.version.outputs.cargo_version }} - # Enable dev-settings feature for test settings (dummy_bool, dummy_int) - # used by e2e tests. - EXTRA_CARGO_FEATURES: openshell-core/dev-settings - run: mise run --no-prepare docker:build:${{ inputs.component }} + run: | + set -euo pipefail + mise exec -- tasks/scripts/docker-build-image.sh "${{ inputs.component }}" \ + --cache-from "type=gha,scope=${{ inputs.component }}-${{ matrix.arch }}" \ + --cache-to "type=gha,mode=max,scope=${{ inputs.component }}-${{ matrix.arch }}" + + - name: Smoke check ${{ inputs.component }} image + if: ${{ !inputs.push }} + run: | + set -euo pipefail + image="${IMAGE_REGISTRY}/${{ inputs.component }}:${IMAGE_TAG}" + case "${{ inputs.component }}" in + gateway) + output="$(docker run --rm --platform "${{ matrix.platform }}" "$image" --version)" + echo "$output" + grep -q '^openshell-gateway ' <<<"$output" + ;; + supervisor) + output="$(docker run --rm --platform "${{ matrix.platform }}" "$image" --version)" + echo "$output" + grep -q '^openshell-sandbox ' <<<"$output" + ;; + cluster) + output="$(docker run --rm --platform "${{ matrix.platform }}" --entrypoint /opt/openshell/bin/openshell-sandbox "$image" --version)" + echo "$output" + grep -q '^openshell-sandbox ' <<<"$output" + ;; + esac + + merge: + name: Merge ${{ inputs.component }} manifest + needs: [resolve, build] + if: ${{ inputs.push && needs.resolve.outputs.platform_count != '1' }} + runs-on: linux-amd64-cpu8 + timeout-minutes: 10 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + volumes: + - /var/run/docker.sock:/var/run/docker.sock + steps: + - name: Log in to GHCR + run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin + + - name: Create multi-arch manifest + run: | + set -euo pipefail + image="ghcr.io/nvidia/openshell/${{ inputs.component }}" + refs=() + for arch in ${{ needs.resolve.outputs.arches }}; do + refs+=("${image}:${{ needs.resolve.outputs.image_tag_base }}-${arch}") + done + docker buildx imagetools create \ + --prefer-index=false \ + -t "${image}:${{ needs.resolve.outputs.image_tag_base }}" \ + "${refs[@]}" diff --git a/.github/workflows/driver-vm-linux.yml b/.github/workflows/driver-vm-linux.yml new file mode 100644 index 0000000000..9647bda05a --- /dev/null +++ b/.github/workflows/driver-vm-linux.yml @@ -0,0 +1,209 @@ +name: Driver VM Linux + +on: + workflow_call: + inputs: + cargo-version: + required: true + type: string + image-tag: + required: true + type: string + checkout-ref: + required: true + type: string + +permissions: + contents: read + packages: read + +defaults: + run: + shell: bash + +jobs: + download-kernel-runtime: + name: Download Kernel Runtime + runs-on: linux-amd64-cpu8 + timeout-minutes: 10 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs['checkout-ref'] }} + + - name: Download Linux runtime tarballs + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + mkdir -p runtime-artifacts + + for platform in linux-aarch64 linux-x86_64; do + asset="vm-runtime-${platform}.tar.zst" + echo "Downloading ${asset}..." + if ! gh release download vm-runtime \ + --repo "${GITHUB_REPOSITORY}" \ + --pattern "${asset}" \ + --dir runtime-artifacts \ + --clobber; then + echo "::error::No ${asset} asset found on vm-runtime release" + exit 1 + fi + done + + ls -lah runtime-artifacts/ + + - name: Verify downloads + run: | + set -euo pipefail + for platform in linux-aarch64 linux-x86_64; do + test -f "runtime-artifacts/vm-runtime-${platform}.tar.zst" + done + + - name: Upload runtime artifacts + uses: actions/upload-artifact@v4 + with: + name: vm-driver-kernel-runtime-tarballs + path: runtime-artifacts/vm-runtime-*.tar.zst + retention-days: 1 + + build-driver-vm-linux: + name: Build Driver VM (Linux ${{ matrix.arch }}) + needs: [download-kernel-runtime] + strategy: + matrix: + include: + - arch: arm64 + runner: linux-arm64-cpu8 + target: aarch64-unknown-linux-gnu + platform: linux-aarch64 + guest_arch: aarch64 + - arch: amd64 + runner: linux-amd64-cpu8 + target: x86_64-unknown-linux-gnu + platform: linux-x86_64 + guest_arch: x86_64 + runs-on: ${{ matrix.runner }} + timeout-minutes: 30 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + options: --privileged + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs['checkout-ref'] }} + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Install tools + run: mise install --locked + + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 + with: + shared-key: driver-vm-linux-${{ matrix.arch }} + cache-directories: .cache/sccache + cache-targets: "true" + + - name: Install zstd + run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/* + + - name: Download kernel runtime tarball + uses: actions/download-artifact@v4 + with: + name: vm-driver-kernel-runtime-tarballs + path: runtime-download/ + + - name: Stage compressed runtime for embedding + run: | + set -euo pipefail + COMPRESSED_DIR="${PWD}/target/vm-runtime-compressed" + mkdir -p "$COMPRESSED_DIR" + + EXTRACT_DIR=$(mktemp -d) + zstd -d "runtime-download/vm-runtime-${{ matrix.platform }}.tar.zst" --stdout \ + | tar -xf - -C "$EXTRACT_DIR" + + echo "Extracted runtime files:" + ls -lah "$EXTRACT_DIR" + + for file in "$EXTRACT_DIR"/*; do + [ -f "$file" ] || continue + name=$(basename "$file") + [ "$name" = "provenance.json" ] && continue + zstd -19 -f -q -T0 -o "${COMPRESSED_DIR}/${name}.zst" "$file" + done + + echo "Staged compressed runtime artifacts:" + ls -lah "$COMPRESSED_DIR" + + - name: Build bundled supervisor + run: | + set -euo pipefail + OPENSHELL_VM_RUNTIME_COMPRESSED_DIR="${PWD}/target/vm-runtime-compressed" \ + tasks/scripts/vm/build-supervisor-bundle.sh --arch "${{ matrix.guest_arch }}" + + - name: Verify embedded driver inputs + run: | + set -euo pipefail + for file in libkrun.so.zst libkrunfw.so.5.zst gvproxy.zst openshell-sandbox.zst; do + test -s "target/vm-runtime-compressed/${file}" + done + + - name: Scope workspace to driver-vm crates + run: | + set -euo pipefail + sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-driver-vm", "crates/openshell-core"]|' Cargo.toml + + - name: Patch workspace version + if: ${{ inputs['cargo-version'] != '' }} + run: | + set -euo pipefail + sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ inputs['cargo-version'] }}"'"/}' Cargo.toml + + - name: Build openshell-driver-vm + run: | + set -euo pipefail + OPENSHELL_VM_RUNTIME_COMPRESSED_DIR="${PWD}/target/vm-runtime-compressed" \ + mise x -- cargo build --release -p openshell-driver-vm + + - name: Verify packaged binary + run: | + set -euo pipefail + OUTPUT="$(target/release/openshell-driver-vm --version)" + echo "$OUTPUT" + grep -q '^openshell-driver-vm ' <<<"$OUTPUT" + + - name: sccache stats + if: always() + run: mise x -- sccache --show-stats + + - name: Package binary + run: | + set -euo pipefail + mkdir -p artifacts + tar -czf "artifacts/openshell-driver-vm-${{ matrix.target }}.tar.gz" \ + -C target/release openshell-driver-vm + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: driver-vm-linux-${{ matrix.arch }} + path: artifacts/*.tar.gz + retention-days: 5 diff --git a/.github/workflows/driver-vm-macos.yml b/.github/workflows/driver-vm-macos.yml new file mode 100644 index 0000000000..cfb041366d --- /dev/null +++ b/.github/workflows/driver-vm-macos.yml @@ -0,0 +1,237 @@ +name: Driver VM macOS + +on: + workflow_call: + inputs: + cargo-version: + required: true + type: string + image-tag: + required: true + type: string + checkout-ref: + required: true + type: string + +permissions: + contents: read + packages: read + +defaults: + run: + shell: bash + +jobs: + download-kernel-runtime: + name: Download Kernel Runtime + runs-on: linux-amd64-cpu8 + timeout-minutes: 10 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs['checkout-ref'] }} + + - name: Download macOS runtime tarball + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + mkdir -p runtime-artifacts + + asset="vm-runtime-darwin-aarch64.tar.zst" + echo "Downloading ${asset}..." + if ! gh release download vm-runtime \ + --repo "${GITHUB_REPOSITORY}" \ + --pattern "${asset}" \ + --dir runtime-artifacts \ + --clobber; then + echo "::error::No ${asset} asset found on vm-runtime release" + exit 1 + fi + + ls -lah runtime-artifacts/ + + - name: Verify download + run: test -f runtime-artifacts/vm-runtime-darwin-aarch64.tar.zst + + - name: Upload runtime artifact + uses: actions/upload-artifact@v4 + with: + name: vm-driver-macos-kernel-runtime-tarball + path: runtime-artifacts/vm-runtime-darwin-aarch64.tar.zst + retention-days: 1 + + build-supervisor-arm64: + name: Build Supervisor Bundle (arm64) + runs-on: linux-arm64-cpu8 + timeout-minutes: 30 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + OPENSHELL_IMAGE_TAG: ${{ inputs['image-tag'] }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs['checkout-ref'] }} + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Install tools + run: mise install --locked + + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 + with: + shared-key: driver-vm-supervisor-arm64 + cache-directories: .cache/sccache + cache-targets: "true" + + - name: Install zstd + run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/* + + - name: Build bundled supervisor + run: | + set -euo pipefail + tasks/scripts/vm/build-supervisor-bundle.sh --arch aarch64 + + - name: sccache stats + if: always() + run: mise x -- sccache --show-stats + + - name: Upload supervisor bundle + uses: actions/upload-artifact@v4 + with: + name: driver-vm-supervisor-arm64 + path: target/vm-runtime-compressed/openshell-sandbox.zst + retention-days: 1 + + build-driver-vm-macos: + name: Build Driver VM (macOS) + needs: [download-kernel-runtime, build-supervisor-arm64] + runs-on: linux-amd64-cpu8 + timeout-minutes: 60 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + options: --privileged + volumes: + - /var/run/docker.sock:/var/run/docker.sock + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs['checkout-ref'] }} + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Log in to GHCR + run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin + + - name: Set up Docker Buildx + uses: ./.github/actions/setup-buildx + with: + driver: local + + - name: Install zstd + run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/* + + - name: Download kernel runtime tarball + uses: actions/download-artifact@v4 + with: + name: vm-driver-macos-kernel-runtime-tarball + path: runtime-download/ + + - name: Prepare compressed runtime directory + run: | + set -euo pipefail + COMPRESSED_DIR="${PWD}/target/vm-runtime-compressed-macos" + mkdir -p "$COMPRESSED_DIR" + + EXTRACT_DIR=$(mktemp -d) + zstd -d "runtime-download/vm-runtime-darwin-aarch64.tar.zst" --stdout \ + | tar -xf - -C "$EXTRACT_DIR" + + echo "Extracted darwin runtime files:" + ls -lah "$EXTRACT_DIR" + + for file in "$EXTRACT_DIR"/*; do + [ -f "$file" ] || continue + name=$(basename "$file") + [ "$name" = "provenance.json" ] && continue + zstd -19 -f -q -T0 -o "${COMPRESSED_DIR}/${name}.zst" "$file" + done + + echo "Staged macOS compressed runtime artifacts:" + ls -lah "$COMPRESSED_DIR" + + - name: Download bundled supervisor + uses: actions/download-artifact@v4 + with: + name: driver-vm-supervisor-arm64 + path: target/vm-runtime-compressed-macos/ + + - name: Verify bundled supervisor + run: | + set -euo pipefail + test -f target/vm-runtime-compressed-macos/openshell-sandbox.zst + ls -lh target/vm-runtime-compressed-macos/openshell-sandbox.zst + + - name: Verify embedded driver inputs + run: | + set -euo pipefail + for file in libkrun.dylib.zst libkrunfw.5.dylib.zst gvproxy.zst openshell-sandbox.zst; do + test -s "target/vm-runtime-compressed-macos/${file}" + done + + - name: Build macOS binary via Docker + run: | + set -euo pipefail + docker buildx build \ + --file deploy/docker/Dockerfile.driver-vm-macos \ + --build-arg OPENSHELL_CARGO_VERSION="${{ inputs['cargo-version'] }}" \ + --build-arg OPENSHELL_IMAGE_TAG="${{ inputs['image-tag'] }}" \ + --build-arg CARGO_TARGET_CACHE_SCOPE="${{ github.sha }}" \ + --build-context vm-runtime-compressed="${PWD}/target/vm-runtime-compressed-macos" \ + --target binary \ + --output type=local,dest=out/ \ + . + + - name: Verify packaged binary shape + run: test -x out/openshell-driver-vm + + - name: Package binary + run: | + set -euo pipefail + mkdir -p artifacts + tar -czf artifacts/openshell-driver-vm-aarch64-apple-darwin.tar.gz \ + -C out openshell-driver-vm + ls -lh artifacts/ + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: driver-vm-macos + path: artifacts/*.tar.gz + retention-days: 5 diff --git a/.github/workflows/e2e-gate-check.yml b/.github/workflows/e2e-gate-check.yml new file mode 100644 index 0000000000..5065663c17 --- /dev/null +++ b/.github/workflows/e2e-gate-check.yml @@ -0,0 +1,113 @@ +name: E2E Gate Check + +# Reusable gate that enforces: when `required_label` is present on a PR, +# `workflow_file` must have completed successfully for the PR head SHA. +# +# Callers wire their own triggers (`pull_request` + `workflow_run` for the +# workflow this gate guards) and pass in the label and workflow filename. + +on: + workflow_call: + inputs: + required_label: + description: PR label that makes the gated workflow mandatory. + required: true + type: string + workflow_file: + description: Filename of the workflow whose run must have succeeded (e.g. "branch-e2e.yml"). + required: true + type: string + +permissions: {} + +jobs: + check: + name: Enforce ${{ inputs.required_label }} runs when labeled + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read + actions: read + steps: + - name: Resolve PR context + id: pr + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GH_REPO: ${{ github.repository }} + EVENT_NAME: ${{ github.event_name }} + PR_HEAD_SHA_FROM_EVENT: ${{ github.event.pull_request.head.sha }} + PR_LABELS_FROM_EVENT: ${{ toJSON(github.event.pull_request.labels.*.name) }} + WORKFLOW_RUN_HEAD_SHA: ${{ github.event.workflow_run.head_sha }} + shell: bash + run: | + set -euo pipefail + if [ "$EVENT_NAME" = "pull_request" ]; then + head_sha="$PR_HEAD_SHA_FROM_EVENT" + labels_json=$(jq -c . <<< "$PR_LABELS_FROM_EVENT") + else + head_sha="$WORKFLOW_RUN_HEAD_SHA" + pr=$(gh api "repos/$GH_REPO/commits/$head_sha/pulls" --jq '.[0] // empty') + if [ -z "$pr" ]; then + echo "No PR associated with $head_sha; gate is a no-op." + echo "skip=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + labels_json=$(jq -c '[.labels[].name]' <<< "$pr") + fi + echo "head_sha=$head_sha" >> "$GITHUB_OUTPUT" + echo "labels_json=$labels_json" >> "$GITHUB_OUTPUT" + + - name: Evaluate gate + if: steps.pr.outputs.skip != 'true' + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GH_REPO: ${{ github.repository }} + HEAD_SHA: ${{ steps.pr.outputs.head_sha }} + LABELS_JSON: ${{ steps.pr.outputs.labels_json }} + REQUIRED_LABEL: ${{ inputs.required_label }} + WORKFLOW_FILE: ${{ inputs.workflow_file }} + shell: bash + run: | + set -euo pipefail + + has_label=$(jq -r --arg L "$REQUIRED_LABEL" 'any(.[]; . == $L)' <<< "$LABELS_JSON") + if [ "$has_label" != "true" ]; then + echo "::notice::$REQUIRED_LABEL not applied; gate passes." + exit 0 + fi + + runs=$(gh api "repos/$GH_REPO/actions/workflows/$WORKFLOW_FILE/runs?head_sha=$HEAD_SHA&event=push" --jq '.workflow_runs') + latest=$(jq -c 'sort_by(.created_at) | reverse | .[0] // empty' <<< "$runs") + + if [ -z "$latest" ]; then + echo "::error::$REQUIRED_LABEL is applied but $WORKFLOW_FILE has not run for $HEAD_SHA. Wait for copy-pr-bot to mirror the PR, or re-run the gate once the workflow completes." + exit 1 + fi + + run_id=$(jq -r '.id' <<< "$latest") + status=$(jq -r '.status' <<< "$latest") + conclusion=$(jq -r '.conclusion' <<< "$latest") + + if [ "$status" != "completed" ]; then + echo "::error::$WORKFLOW_FILE is $status for $HEAD_SHA. This gate will re-evaluate on completion." + exit 1 + fi + + if [ "$conclusion" != "success" ]; then + echo "::error::$WORKFLOW_FILE concluded as $conclusion for $HEAD_SHA." + exit 1 + fi + + # Top-level success isn't enough: if `pr_metadata` gated downstream + # jobs out (label wasn't set at run time), only the gate job itself + # concludes `success` and the workflow still reports `success`. + # Require at least one non-gate job to have succeeded as proof the + # label was present when the workflow ran. + real_success=$(gh api "repos/$GH_REPO/actions/runs/$run_id/jobs" --jq '[.jobs[] | select(.conclusion == "success" and .name != "Resolve PR metadata")] | length') + if [ "$real_success" -lt 1 ]; then + echo "::error::$WORKFLOW_FILE run $run_id only ran the metadata gate — $REQUIRED_LABEL was not set when the workflow last executed. Re-run $WORKFLOW_FILE so the gate re-evaluates with the label present." + exit 1 + fi + + echo "$WORKFLOW_FILE run $run_id executed and succeeded for $HEAD_SHA ($real_success non-gate job(s) passed)." + exit 0 diff --git a/.github/workflows/e2e-gate.yml b/.github/workflows/e2e-gate.yml new file mode 100644 index 0000000000..67959fa8dc --- /dev/null +++ b/.github/workflows/e2e-gate.yml @@ -0,0 +1,67 @@ +name: E2E Gate + +on: + pull_request: + types: [opened, synchronize, reopened, labeled, unlabeled, ready_for_review] + workflow_run: + workflows: ["Branch E2E Checks", "GPU Test"] + types: [completed] + +permissions: {} + +jobs: + # On PR events, actually evaluate the gate — these runs post their check + # result to the PR's head SHA, which is what branch protection sees. + e2e: + name: E2E + if: github.event_name == 'pull_request' + permissions: + contents: read + pull-requests: read + actions: read + uses: ./.github/workflows/e2e-gate-check.yml + with: + required_label: test:e2e + workflow_file: branch-e2e.yml + + gpu: + name: GPU E2E + if: github.event_name == 'pull_request' + permissions: + contents: read + pull-requests: read + actions: read + uses: ./.github/workflows/e2e-gate-check.yml + with: + required_label: test:e2e-gpu + workflow_file: test-gpu.yml + + # When the guarded workflow finishes, GitHub fires `workflow_run` in the + # default-branch context — any check posted from here would land on `main`, + # not on the PR. Instead, find the latest `pull_request`-triggered gate run + # for the same head SHA and ask the API to re-run it. The re-run replays the + # original event (labels, head SHA) so the check posts to the PR again, and + # this time the gate sees the successful upstream run. + rerun-on-completion: + name: Re-run gate in PR context + if: github.event_name == 'workflow_run' + runs-on: ubuntu-latest + permissions: + actions: write + steps: + - name: Rerun latest PR-context gate for this SHA + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GH_REPO: ${{ github.repository }} + HEAD_SHA: ${{ github.event.workflow_run.head_sha }} + shell: bash + run: | + set -euo pipefail + run_id=$(gh api "repos/$GH_REPO/actions/workflows/e2e-gate.yml/runs?event=pull_request&head_sha=$HEAD_SHA" \ + --jq '.workflow_runs | sort_by(.created_at) | reverse | .[0].id // empty') + if [ -z "$run_id" ]; then + echo "No pull_request-triggered E2E Gate run found for $HEAD_SHA — nothing to re-run." + exit 0 + fi + echo "Re-running E2E Gate run $run_id so it re-evaluates and posts a fresh check on the PR." + gh api --method POST "repos/$GH_REPO/actions/runs/$run_id/rerun" diff --git a/.github/workflows/e2e-gpu-test.yaml b/.github/workflows/e2e-gpu-test.yaml index 9feda37695..6a296f5e30 100644 --- a/.github/workflows/e2e-gpu-test.yaml +++ b/.github/workflows/e2e-gpu-test.yaml @@ -55,7 +55,7 @@ jobs: OPENSHELL_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }} OPENSHELL_GATEWAY: ${{ matrix.cluster }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Log in to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin @@ -64,7 +64,7 @@ jobs: run: docker pull ghcr.io/nvidia/openshell/cluster:${{ inputs.image-tag }} - name: Install Python dependencies and generate protobuf stubs - run: uv sync --frozen && mise run --no-prepare python:proto + run: uv sync --frozen && mise run --no-deps python:proto - name: Bootstrap GPU cluster env: @@ -76,7 +76,7 @@ jobs: SKIP_IMAGE_PUSH: "1" SKIP_CLUSTER_IMAGE_BUILD: "1" OPENSHELL_CLUSTER_IMAGE: ghcr.io/nvidia/openshell/cluster:${{ inputs.image-tag }} - run: mise run --no-prepare --skip-deps cluster + run: mise run --no-deps --skip-deps cluster - name: Run tests - run: mise run --no-prepare --skip-deps e2e:python:gpu + run: mise run --no-deps --skip-deps e2e:python:gpu diff --git a/.github/workflows/e2e-label-help.yml b/.github/workflows/e2e-label-help.yml new file mode 100644 index 0000000000..2a61660d2a --- /dev/null +++ b/.github/workflows/e2e-label-help.yml @@ -0,0 +1,67 @@ +name: E2E Label Help + +# When a `test:e2e` / `test:e2e-gpu` label is applied, post a PR comment +# telling the maintainer the next manual step. We don't dispatch the workflow +# ourselves: a workflow_dispatch-triggered run does not surface in the PR's +# Checks tab, so we'd lose in-progress visibility. Instead we point the +# maintainer at either the existing run (re-run from the UI) or the +# `/ok to test ` command needed to refresh the mirror. +# +# Uses `pull_request_target` so forked PRs get a token capable of posting +# comments. The job never checks out PR code; it only calls the GitHub API. + +on: + pull_request_target: + types: [labeled] + +permissions: {} + +jobs: + hint: + name: Post next-step hint for E2E label + if: github.event.label.name == 'test:e2e' || github.event.label.name == 'test:e2e-gpu' + runs-on: ubuntu-latest + permissions: + pull-requests: write + actions: read + contents: read + steps: + - name: Post comment + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GH_REPO: ${{ github.repository }} + PR_NUMBER: ${{ github.event.pull_request.number }} + PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + LABEL_NAME: ${{ github.event.label.name }} + shell: bash + run: | + set -euo pipefail + + case "$LABEL_NAME" in + test:e2e) workflow_file=branch-e2e.yml; workflow_name="Branch E2E Checks" ;; + test:e2e-gpu) workflow_file=test-gpu.yml; workflow_name="GPU Test" ;; + *) echo "Unrecognized label $LABEL_NAME"; exit 1 ;; + esac + + mirror_ref="pull-request/$PR_NUMBER" + mirror_sha=$(gh api "repos/$GH_REPO/branches/$mirror_ref" --jq '.commit.sha' 2>/dev/null || echo "") + short_pr=${PR_HEAD_SHA:0:7} + + if [ -z "$mirror_sha" ]; then + body="Label \`$LABEL_NAME\` applied, but \`$mirror_ref\` does not exist yet. A maintainer needs to comment \`/ok to test $PR_HEAD_SHA\` to mirror this PR. Once the mirror exists, re-apply the label or re-run [$workflow_name](https://github.com/$GH_REPO/actions/workflows/$workflow_file) from the Actions tab." + elif [ "$mirror_sha" != "$PR_HEAD_SHA" ]; then + short_mirror=${mirror_sha:0:7} + body="Label \`$LABEL_NAME\` applied, but \`$mirror_ref\` is at \`$short_mirror\` while the PR head is \`$short_pr\`. A maintainer needs to comment \`/ok to test $PR_HEAD_SHA\` to refresh the mirror. Once the mirror catches up, re-run [$workflow_name](https://github.com/$GH_REPO/actions/workflows/$workflow_file) from the Actions tab." + else + run_id=$(gh api "repos/$GH_REPO/actions/workflows/$workflow_file/runs?head_sha=$PR_HEAD_SHA&event=push" \ + --jq '.workflow_runs | sort_by(.created_at) | reverse | .[0].id // empty') + if [ -n "$run_id" ]; then + instructions="Open [the existing run](https://github.com/$GH_REPO/actions/runs/$run_id) and click **Re-run all jobs** to execute with the label set." + else + workflow_link="[$workflow_name](https://github.com/$GH_REPO/actions/workflows/$workflow_file)" + instructions="Open $workflow_link, find the run for commit \`$short_pr\`, and click **Re-run all jobs** to execute with the label set." + fi + body="Label \`$LABEL_NAME\` applied for \`$short_pr\`. $instructions The \`E2E Gate\` check on this PR will flip green automatically once the run finishes." + fi + + gh pr comment "$PR_NUMBER" --body "$body" diff --git a/.github/workflows/e2e-test.yml b/.github/workflows/e2e-test.yml index a89f4508fc..536cfb2a50 100644 --- a/.github/workflows/e2e-test.yml +++ b/.github/workflows/e2e-test.yml @@ -11,7 +11,7 @@ on: description: "GitHub Actions runner label" required: false type: string - default: "build-amd64" + default: "linux-amd64-cpu8" permissions: contents: read @@ -27,17 +27,9 @@ jobs: matrix: include: - suite: python - cluster: e2e-python - port: "8080" - cmd: "mise run --no-prepare --skip-deps e2e:python" + cmd: "mise run --no-deps --skip-deps e2e:python" - suite: rust - cluster: e2e-rust - port: "8081" - cmd: "mise run --no-prepare --skip-deps e2e:rust" - - suite: gateway-resume - cluster: e2e-resume - port: "8082" - cmd: "cargo test --manifest-path e2e/rust/Cargo.toml --features e2e --test gateway_resume" + cmd: "mise run --no-deps --skip-deps e2e:rust" container: image: ghcr.io/nvidia/openshell/ci:latest credentials: @@ -46,6 +38,7 @@ jobs: options: --privileged volumes: - /var/run/docker.sock:/var/run/docker.sock + - /home/runner/_work:/home/runner/_work env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} IMAGE_TAG: ${{ inputs.image-tag }} @@ -54,37 +47,19 @@ jobs: OPENSHELL_REGISTRY_NAMESPACE: nvidia/openshell OPENSHELL_REGISTRY_USERNAME: ${{ github.actor }} OPENSHELL_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }} - OPENSHELL_GATEWAY: ${{ matrix.cluster }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Log in to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin - - name: Pull cluster image - run: docker pull ghcr.io/nvidia/openshell/cluster:${{ inputs.image-tag }} - - name: Install Python dependencies and generate protobuf stubs if: matrix.suite == 'python' - run: uv sync --frozen && mise run --no-prepare python:proto - - - name: Build Rust CLI - if: matrix.suite != 'python' - run: cargo build -p openshell-cli --features openshell-core/dev-settings + run: uv sync --frozen && mise run --no-deps python:proto - name: Install SSH client if: matrix.suite != 'python' run: apt-get update && apt-get install -y --no-install-recommends openssh-client && rm -rf /var/lib/apt/lists/* - - name: Bootstrap cluster - env: - GATEWAY_HOST: host.docker.internal - GATEWAY_PORT: ${{ matrix.port }} - CLUSTER_NAME: ${{ matrix.cluster }} - SKIP_IMAGE_PUSH: "1" - SKIP_CLUSTER_IMAGE_BUILD: "1" - OPENSHELL_CLUSTER_IMAGE: ghcr.io/nvidia/openshell/cluster:${{ inputs.image-tag }} - run: mise run --no-prepare --skip-deps cluster - - name: Run tests run: ${{ matrix.cmd }} diff --git a/.github/workflows/issue-triage.yml b/.github/workflows/issue-triage.yml index ec87af503c..2549172596 100644 --- a/.github/workflows/issue-triage.yml +++ b/.github/workflows/issue-triage.yml @@ -8,53 +8,50 @@ permissions: issues: write jobs: - check-agent-diagnostic: + label-non-maintainer-issues: runs-on: ubuntu-latest - # Only run on bug reports (title starts with "bug:") - if: startsWith(github.event.issue.title, 'bug:') + if: github.repository_owner == 'NVIDIA' steps: - - name: Check for agent diagnostic section + - name: Check contributor permissions + id: contributor uses: actions/github-script@v7 with: + result-encoding: string script: | - const body = context.payload.issue.body || ''; - - // Check if the Agent Diagnostic section has substantive content. - // The template placeholder starts with "Example:" — if that's still - // there or the section is empty, the reporter didn't fill it in. - const diagnosticMatch = body.match( - /### Agent Diagnostic\s*\n([\s\S]*?)(?=\n### |\n$)/ - ); - - const hasSubstantiveDiagnostic = diagnosticMatch - && diagnosticMatch[1].trim().length > 0 - && !diagnosticMatch[1].trim().startsWith('Example:'); - - if (hasSubstantiveDiagnostic) { - console.log('Agent diagnostic section found with content. Passing.'); - return; + const author = context.payload.issue.user.login; + const maintainerPermissions = ['admin', 'maintain', 'write']; + + try { + const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, + repo: context.repo.repo, + username: author, + }); + + if (maintainerPermissions.includes(data.permission)) { + console.log(`${author} has maintainer permissions: ${data.permission}.`); + return 'false'; + } + + console.log(`${author} is not a maintainer; permission is ${data.permission}.`); + return 'true'; + } catch (e) { + if (e.status === 404) { + console.log(`${author} is not a repository collaborator; labeling for triage.`); + return 'true'; + } + + throw e; } - console.log('Agent diagnostic section missing or placeholder. Flagging.'); - - // Add state:triage-needed label + - name: Add triage label + if: steps.contributor.outputs.result == 'true' + uses: actions/github-script@v7 + with: + script: | await github.rest.issues.addLabels({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.issue.number, - labels: ['state:triage-needed'] - }); - - // Post redirect comment - await github.rest.issues.createComment({ - owner: context.repo.owner, - repo: context.repo.repo, - issue_number: context.issue.number, - body: [ - 'This issue appears to have been opened without an agent investigation.', - '', - 'OpenShell is an agent-first project - please point your coding agent at the repo and have it diagnose this before we triage. Your agent can load skills like `debug-openshell-cluster`, `debug-inference`, `openshell-cli`, and `generate-sandbox-policy`.', - '', - 'See [CONTRIBUTING.md](https://github.com/NVIDIA/OpenShell/blob/main/CONTRIBUTING.md#before-you-open-an-issue) for the full workflow.', - ].join('\n') + labels: ['state:triage-needed'], }); diff --git a/.github/workflows/release-auto-tag.yml b/.github/workflows/release-auto-tag.yml index 569c333887..f89c506d7b 100644 --- a/.github/workflows/release-auto-tag.yml +++ b/.github/workflows/release-auto-tag.yml @@ -20,7 +20,7 @@ jobs: create-tag: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 diff --git a/.github/workflows/release-canary.yml b/.github/workflows/release-canary.yml index 0ef1e0355e..8f7284aabf 100644 --- a/.github/workflows/release-canary.yml +++ b/.github/workflows/release-canary.yml @@ -33,9 +33,9 @@ jobs: matrix: include: - arch: amd64 - runner: build-amd64 + runner: linux-amd64-cpu8 - arch: arm64 - runner: build-arm64 + runner: linux-arm64-cpu8 runs-on: ${{ matrix.runner }} timeout-minutes: 10 container: @@ -67,6 +67,79 @@ jobs: echo "Version check passed: found $EXPECTED in output" fi + install-dev: + name: Install Debian package (${{ matrix.arch }}) + if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} + strategy: + fail-fast: false + matrix: + include: + - arch: amd64 + runner: linux-amd64-cpu8 + - arch: arm64 + runner: linux-arm64-cpu8 + runs-on: ${{ matrix.runner }} + timeout-minutes: 10 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: Determine release tag + id: release + run: | + set -euo pipefail + if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + echo "tag=${{ inputs.tag }}" >> "$GITHUB_OUTPUT" + else + WORKFLOW_NAME="${{ github.event.workflow_run.name }}" + if [ "$WORKFLOW_NAME" = "Release Dev" ]; then + echo "tag=dev" >> "$GITHUB_OUTPUT" + elif [ "$WORKFLOW_NAME" = "Release Tag" ]; then + TAG="${{ github.event.workflow_run.head_branch }}" + if [ -z "$TAG" ]; then + echo "::error::Could not determine release tag from workflow_run" + exit 1 + fi + echo "tag=${TAG}" >> "$GITHUB_OUTPUT" + else + echo "::error::Unexpected triggering workflow: ${WORKFLOW_NAME}" + exit 1 + fi + fi + + - name: Install Debian package + run: | + set -euo pipefail + curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install-dev.sh \ + | OPENSHELL_VERSION=${{ steps.release.outputs.tag }} sh + + - name: Verify gateway and VM driver versions + run: | + set -euo pipefail + command -v openshell-gateway + test -x /usr/libexec/openshell/openshell-driver-vm + + GATEWAY_ACTUAL="$(openshell-gateway --version)" + DRIVER_ACTUAL="$(/usr/libexec/openshell/openshell-driver-vm --version)" + echo "Gateway: ${GATEWAY_ACTUAL}" + echo "Driver: ${DRIVER_ACTUAL}" + + TAG="${{ steps.release.outputs.tag }}" + if [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + EXPECTED="${TAG#v}" + for actual in "$GATEWAY_ACTUAL" "$DRIVER_ACTUAL"; do + if [[ "$actual" != *"$EXPECTED"* ]]; then + echo "::error::Version mismatch: expected '$EXPECTED' in '$actual'" + exit 1 + fi + done + echo "Version check passed: found $EXPECTED in both binaries" + else + echo "Non-release tag ($TAG), skipping version check" + fi + canary: name: Canary ${{ matrix.mode }} (${{ matrix.arch }}) if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} @@ -81,10 +154,10 @@ jobs: - two-step include: - arch: amd64 - runner: build-amd64 + runner: linux-amd64-cpu8 target: x86_64-unknown-linux-musl - arch: arm64 - runner: build-arm64 + runner: linux-arm64-cpu8 target: aarch64-unknown-linux-musl runs-on: ${{ matrix.runner }} timeout-minutes: 30 @@ -104,7 +177,7 @@ jobs: # to advertise a reachable address instead. OPENSHELL_GATEWAY_HOST: host.docker.internal steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Determine release tag id: release diff --git a/.github/workflows/release-dev.yml b/.github/workflows/release-dev.yml index bdc8b1a719..a359dde599 100644 --- a/.github/workflows/release-dev.yml +++ b/.github/workflows/release-dev.yml @@ -19,7 +19,7 @@ jobs: # --------------------------------------------------------------------------- compute-versions: name: Compute Versions - runs-on: build-amd64 + runs-on: linux-amd64-cpu8 timeout-minutes: 5 container: image: ghcr.io/nvidia/openshell/ci:latest @@ -29,8 +29,9 @@ jobs: outputs: python_version: ${{ steps.v.outputs.python }} cargo_version: ${{ steps.v.outputs.cargo }} + deb_version: ${{ steps.v.outputs.deb }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 @@ -46,6 +47,7 @@ jobs: set -euo pipefail echo "python=$(uv run python tasks/scripts/release.py get-version --python)" >> "$GITHUB_OUTPUT" echo "cargo=$(uv run python tasks/scripts/release.py get-version --cargo)" >> "$GITHUB_OUTPUT" + echo "deb=$(uv run python tasks/scripts/release.py get-version --deb)" >> "$GITHUB_OUTPUT" build-gateway: needs: [compute-versions] @@ -54,6 +56,13 @@ jobs: component: gateway cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + build-supervisor: + needs: [compute-versions] + uses: ./.github/workflows/docker-build.yml + with: + component: supervisor + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + build-cluster: needs: [compute-versions] uses: ./.github/workflows/docker-build.yml @@ -66,12 +75,12 @@ jobs: uses: ./.github/workflows/e2e-test.yml with: image-tag: ${{ github.sha }} - runner: build-arm64 + runner: linux-arm64-cpu8 tag-ghcr-dev: name: Tag GHCR Images as Dev - needs: [build-gateway, build-cluster] - runs-on: build-amd64 + needs: [build-gateway, build-supervisor, build-cluster] + runs-on: linux-amd64-cpu8 timeout-minutes: 10 steps: - name: Log in to GHCR @@ -81,7 +90,7 @@ jobs: run: | set -euo pipefail REGISTRY="ghcr.io/nvidia/openshell" - for component in gateway cluster; do + for component in gateway supervisor cluster; do echo "Tagging ${REGISTRY}/${component}:${{ github.sha }} as dev..." docker buildx imagetools create \ --prefer-index=false \ @@ -96,12 +105,12 @@ jobs: matrix: include: - arch: amd64 - runner: build-amd64 + runner: linux-amd64-cpu8 artifact: linux-amd64 task: python:build:linux:amd64 output_path: target/wheels/linux-amd64/*.whl - arch: arm64 - runner: build-arm64 + runner: linux-arm64-cpu8 artifact: linux-arm64 task: python:build:linux:arm64 output_path: target/wheels/linux-arm64/*.whl @@ -115,10 +124,9 @@ jobs: options: --privileged env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} OPENSHELL_IMAGE_TAG: dev steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 @@ -151,7 +159,7 @@ jobs: build-python-wheel-macos: name: Build Python Wheel (macOS) needs: [compute-versions] - runs-on: build-amd64 + runs-on: linux-amd64-cpu8 timeout-minutes: 120 container: image: ghcr.io/nvidia/openshell/ci:latest @@ -163,10 +171,9 @@ jobs: - /var/run/docker.sock:/var/run/docker.sock env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} OPENSHELL_IMAGE_TAG: dev steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 @@ -175,6 +182,8 @@ jobs: - name: Set up Docker Buildx uses: ./.github/actions/setup-buildx + with: + driver: local - name: Mark workspace safe for git run: git config --global --add safe.directory "$GITHUB_WORKSPACE" @@ -208,11 +217,11 @@ jobs: matrix: include: - arch: amd64 - runner: build-amd64 + runner: linux-amd64-cpu8 target: x86_64-unknown-linux-musl zig_target: x86_64-linux-musl - arch: arm64 - runner: build-arm64 + runner: linux-arm64-cpu8 target: aarch64-unknown-linux-musl zig_target: aarch64-linux-musl runs-on: ${{ matrix.runner }} @@ -225,10 +234,9 @@ jobs: options: --privileged env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} OPENSHELL_IMAGE_TAG: dev steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 @@ -239,7 +247,7 @@ jobs: run: git fetch --tags --force - name: Install tools - run: mise install + run: mise install --locked - name: Cache Rust target and registry uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 @@ -282,11 +290,6 @@ jobs: # Override z3-sys default (stdc++) so Rust links the matching runtime. echo "CXXSTDLIB=c++" >> "$GITHUB_ENV" - - name: Scope workspace to CLI crates - run: | - set -euo pipefail - sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-cli", "crates/openshell-core", "crates/openshell-bootstrap", "crates/openshell-policy", "crates/openshell-prover", "crates/openshell-providers", "crates/openshell-tui"]|' Cargo.toml - - name: Patch workspace version if: needs.compute-versions.outputs.cargo_version != '' run: | @@ -321,7 +324,7 @@ jobs: build-cli-macos: name: Build CLI (macOS) needs: [compute-versions] - runs-on: build-amd64 + runs-on: linux-amd64-cpu8 timeout-minutes: 60 container: image: ghcr.io/nvidia/openshell/ci:latest @@ -333,9 +336,8 @@ jobs: - /var/run/docker.sock:/var/run/docker.sock env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: fetch-depth: 0 @@ -350,6 +352,8 @@ jobs: - name: Set up Docker Buildx uses: ./.github/actions/setup-buildx + with: + driver: local - name: Build macOS binary via Docker run: | @@ -379,17 +383,293 @@ jobs: retention-days: 5 # --------------------------------------------------------------------------- - # Create / update the dev GitHub Release with CLI binaries and wheels + # Build standalone gateway binaries (Linux GNU — native on each arch) + # --------------------------------------------------------------------------- + build-gateway-binary-linux: + name: Build Gateway Binary (Linux ${{ matrix.arch }}) + needs: [compute-versions] + strategy: + matrix: + include: + - arch: amd64 + runner: linux-amd64-cpu8 + target: x86_64-unknown-linux-gnu + - arch: arm64 + runner: linux-arm64-cpu8 + target: aarch64-unknown-linux-gnu + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + options: --privileged + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Install tools + run: mise install --locked + + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 + with: + shared-key: gateway-binary-gnu-${{ matrix.arch }} + cache-directories: .cache/sccache + cache-targets: "true" + + - name: Patch workspace version + if: needs.compute-versions.outputs.cargo_version != '' + run: | + set -euo pipefail + sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ needs.compute-versions.outputs.cargo_version }}"'"/}' Cargo.toml + + - name: Build ${{ matrix.target }} + run: | + set -euo pipefail + mise x -- cargo build --release --target ${{ matrix.target }} -p openshell-server + + - name: Verify packaged binary + run: | + set -euo pipefail + OUTPUT="$(target/${{ matrix.target }}/release/openshell-gateway --version)" + echo "$OUTPUT" + grep -q '^openshell-gateway ' <<<"$OUTPUT" + + - name: sccache stats + if: always() + run: mise x -- sccache --show-stats + + - name: Package binary + run: | + set -euo pipefail + mkdir -p artifacts + tar -czf artifacts/openshell-gateway-${{ matrix.target }}.tar.gz \ + -C target/${{ matrix.target }}/release openshell-gateway + ls -lh artifacts/ + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: gateway-binary-linux-${{ matrix.arch }} + path: artifacts/*.tar.gz + retention-days: 5 + + # --------------------------------------------------------------------------- + # Build standalone gateway binary (macOS aarch64 via osxcross) + # --------------------------------------------------------------------------- + build-gateway-binary-macos: + name: Build Gateway Binary (macOS) + needs: [compute-versions] + runs-on: linux-amd64-cpu8 + timeout-minutes: 60 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + options: --privileged + volumes: + - /var/run/docker.sock:/var/run/docker.sock + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Log in to GHCR + run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin + + - name: Set up Docker Buildx + uses: ./.github/actions/setup-buildx + with: + driver: local + + - name: Build macOS binary via Docker + run: | + set -euo pipefail + docker buildx build \ + --file deploy/docker/Dockerfile.gateway-macos \ + --build-arg OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \ + --build-arg CARGO_TARGET_CACHE_SCOPE="${{ github.sha }}" \ + --target binary \ + --output type=local,dest=out/ \ + . + + - name: Verify packaged binary shape + run: | + set -euo pipefail + test -x out/openshell-gateway + + - name: Package binary + run: | + set -euo pipefail + mkdir -p artifacts + tar -czf artifacts/openshell-gateway-aarch64-apple-darwin.tar.gz \ + -C out openshell-gateway + ls -lh artifacts/ + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: gateway-binary-macos + path: artifacts/*.tar.gz + retention-days: 5 + + # --------------------------------------------------------------------------- + # Build standalone supervisor binaries (Linux GNU — native on each arch) + # --------------------------------------------------------------------------- + build-supervisor-binary-linux: + name: Build Supervisor Binary (Linux ${{ matrix.arch }}) + needs: [compute-versions] + strategy: + matrix: + include: + - arch: amd64 + runner: linux-amd64-cpu8 + target: x86_64-unknown-linux-gnu + - arch: arm64 + runner: linux-arm64-cpu8 + target: aarch64-unknown-linux-gnu + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + options: --privileged + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Install tools + run: mise install --locked + + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 + with: + shared-key: supervisor-binary-gnu-${{ matrix.arch }} + cache-directories: .cache/sccache + cache-targets: "true" + + - name: Patch workspace version + if: needs.compute-versions.outputs.cargo_version != '' + run: | + set -euo pipefail + sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ needs.compute-versions.outputs.cargo_version }}"'"/}' Cargo.toml + + - name: Build ${{ matrix.target }} + run: | + set -euo pipefail + mise x -- cargo build --release --target ${{ matrix.target }} -p openshell-sandbox --bin openshell-sandbox + + - name: Verify packaged binary + run: | + set -euo pipefail + OUTPUT="$(target/${{ matrix.target }}/release/openshell-sandbox --version)" + echo "$OUTPUT" + grep -q '^openshell-sandbox ' <<<"$OUTPUT" + + - name: sccache stats + if: always() + run: mise x -- sccache --show-stats + + - name: Package binary + run: | + set -euo pipefail + mkdir -p artifacts + tar -czf artifacts/openshell-sandbox-${{ matrix.target }}.tar.gz \ + -C target/${{ matrix.target }}/release openshell-sandbox + ls -lh artifacts/ + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: supervisor-binary-linux-${{ matrix.arch }} + path: artifacts/*.tar.gz + retention-days: 5 + + build-driver-vm-linux: + name: Build Driver VM Linux + needs: [compute-versions] + uses: ./.github/workflows/driver-vm-linux.yml + with: + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + image-tag: dev + checkout-ref: ${{ github.sha }} + secrets: inherit + + build-driver-vm-macos: + name: Build Driver VM macOS + needs: [compute-versions] + uses: ./.github/workflows/driver-vm-macos.yml + with: + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + image-tag: dev + checkout-ref: ${{ github.sha }} + secrets: inherit + + build-deb: + name: Build Debian Packages + needs: [compute-versions, build-cli-linux, build-gateway-binary-linux, build-driver-vm-linux] + uses: ./.github/workflows/deb-package.yml + with: + deb-version: ${{ needs.compute-versions.outputs.deb_version }} + checkout-ref: ${{ github.sha }} + secrets: inherit + + build-rpm: + name: Build RPM Packages + needs: [compute-versions] + uses: ./.github/workflows/rpm-package.yml + with: + checkout-ref: ${{ github.sha }} + secrets: inherit + + # --------------------------------------------------------------------------- + # Create / update the dev GitHub Release with CLI, gateway, driver, and wheels # --------------------------------------------------------------------------- release-dev: name: Release Dev - needs: [compute-versions, build-cli-linux, build-cli-macos, build-python-wheels-linux, build-python-wheel-macos] - runs-on: build-amd64 + needs: [compute-versions, build-cli-linux, build-cli-macos, build-gateway-binary-linux, build-gateway-binary-macos, build-supervisor-binary-linux, build-python-wheels-linux, build-python-wheel-macos, build-driver-vm-linux, build-driver-vm-macos, build-deb, build-rpm] + runs-on: linux-amd64-cpu8 timeout-minutes: 10 + permissions: + contents: write + id-token: write + attestations: write + artifact-metadata: write outputs: wheel_filenames: ${{ steps.wheel_filenames.outputs.wheel_filenames }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Download all CLI artifacts uses: actions/download-artifact@v4 @@ -398,6 +678,27 @@ jobs: path: release/ merge-multiple: true + - name: Download gateway binary artifacts + uses: actions/download-artifact@v4 + with: + pattern: gateway-binary-* + path: release/ + merge-multiple: true + + - name: Download supervisor binary artifacts + uses: actions/download-artifact@v4 + with: + pattern: supervisor-binary-* + path: release/ + merge-multiple: true + + - name: Download VM driver artifacts + uses: actions/download-artifact@v4 + with: + pattern: driver-vm-* + path: release/ + merge-multiple: true + - name: Download wheel artifacts uses: actions/download-artifact@v4 with: @@ -405,6 +706,20 @@ jobs: path: release/ merge-multiple: true + - name: Download Debian package artifacts + uses: actions/download-artifact@v4 + with: + pattern: deb-linux-* + path: release/ + merge-multiple: true + + - name: Download RPM package artifacts + uses: actions/download-artifact@v4 + with: + pattern: rpm-linux-* + path: release/ + merge-multiple: true + - name: Capture wheel filenames id: wheel_filenames run: | @@ -417,10 +732,33 @@ jobs: run: | set -euo pipefail cd release - sha256sum *.tar.gz *.whl > openshell-checksums-sha256.txt + sha256sum \ + openshell-x86_64-unknown-linux-musl.tar.gz \ + openshell-aarch64-unknown-linux-musl.tar.gz \ + openshell-aarch64-apple-darwin.tar.gz \ + openshell_*.deb \ + openshell-*.rpm \ + *.whl > openshell-checksums-sha256.txt cat openshell-checksums-sha256.txt + sha256sum \ + openshell-gateway-x86_64-unknown-linux-gnu.tar.gz \ + openshell-gateway-aarch64-unknown-linux-gnu.tar.gz \ + openshell-gateway-aarch64-apple-darwin.tar.gz > openshell-gateway-checksums-sha256.txt + cat openshell-gateway-checksums-sha256.txt + sha256sum \ + openshell-sandbox-x86_64-unknown-linux-gnu.tar.gz \ + openshell-sandbox-aarch64-unknown-linux-gnu.tar.gz > openshell-sandbox-checksums-sha256.txt + cat openshell-sandbox-checksums-sha256.txt + + - name: Attest VM driver artifacts + uses: actions/attest@v4 + with: + subject-path: | + release/openshell-driver-vm-x86_64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-aarch64-apple-darwin.tar.gz - - name: Prune stale wheel assets from dev release + - name: Prune stale wheel, deb, and rpm assets from dev release uses: actions/github-script@v7 env: WHEEL_VERSION: ${{ needs.compute-versions.outputs.python_version }} @@ -452,20 +790,31 @@ jobs: core.info(` ${String(a.id).padStart(12)} ${a.name}`); } - // Delete stale wheels - let kept = 0, deleted = 0; + // Delete stale wheels, debs, rpms, and removed VM checksum assets. + let kept = 0, deleted = 0, debDeleted = 0, rpmDeleted = 0, removedVmChecksums = 0; for (const asset of assets) { - if (!asset.name.endsWith('.whl')) continue; - if (asset.name.startsWith(currentPrefix)) { + if (asset.name.endsWith('.deb')) { + core.info(`Deleting stale deb package: ${asset.name} (id=${asset.id})`); + await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: asset.id }); + debDeleted++; + } else if (asset.name.endsWith('.rpm')) { + core.info(`Deleting stale rpm package: ${asset.name} (id=${asset.id})`); + await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: asset.id }); + rpmDeleted++; + } else if (asset.name === 'openshell-driver-vm-checksums-sha256.txt') { + core.info(`Deleting removed VM checksum asset: ${asset.name} (id=${asset.id})`); + await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: asset.id }); + removedVmChecksums++; + } else if (asset.name.endsWith('.whl') && asset.name.startsWith(currentPrefix)) { core.info(`Keeping current wheel: ${asset.name}`); kept++; - } else { + } else if (asset.name.endsWith('.whl')) { core.info(`Deleting stale wheel: ${asset.name} (id=${asset.id})`); await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: asset.id }); deleted++; } } - core.info(`Summary: kept=${kept}, deleted=${deleted}`); + core.info(`Summary: kept_wheels=${kept}, deleted_wheels=${deleted}, deleted_debs=${debDeleted}, deleted_rpms=${rpmDeleted}, deleted_removed_vm_checksums=${removedVmChecksums}`); - name: Move dev tag run: | @@ -496,8 +845,20 @@ jobs: release/openshell-x86_64-unknown-linux-musl.tar.gz release/openshell-aarch64-unknown-linux-musl.tar.gz release/openshell-aarch64-apple-darwin.tar.gz + release/openshell_*.deb + release/openshell-*.rpm + release/openshell-gateway-x86_64-unknown-linux-gnu.tar.gz + release/openshell-gateway-aarch64-unknown-linux-gnu.tar.gz + release/openshell-gateway-aarch64-apple-darwin.tar.gz + release/openshell-sandbox-x86_64-unknown-linux-gnu.tar.gz + release/openshell-sandbox-aarch64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-x86_64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-aarch64-apple-darwin.tar.gz release/*.whl release/openshell-checksums-sha256.txt + release/openshell-gateway-checksums-sha256.txt + release/openshell-sandbox-checksums-sha256.txt trigger-wheel-publish: name: Trigger Wheel Publish diff --git a/.github/workflows/release-tag.yml b/.github/workflows/release-tag.yml index d89d67032b..b8abb5a899 100644 --- a/.github/workflows/release-tag.yml +++ b/.github/workflows/release-tag.yml @@ -30,7 +30,7 @@ jobs: # --------------------------------------------------------------------------- compute-versions: name: Compute Versions - runs-on: build-amd64 + runs-on: linux-amd64-cpu8 timeout-minutes: 5 container: image: ghcr.io/nvidia/openshell/ci:latest @@ -40,10 +40,11 @@ jobs: outputs: python_version: ${{ steps.v.outputs.python }} cargo_version: ${{ steps.v.outputs.cargo }} + deb_version: ${{ steps.v.outputs.deb }} # Semver without 'v' prefix (e.g. 0.6.0), used for image tags and release body semver: ${{ steps.v.outputs.semver }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -60,6 +61,7 @@ jobs: set -euo pipefail echo "python=$(uv run python tasks/scripts/release.py get-version --python)" >> "$GITHUB_OUTPUT" echo "cargo=$(uv run python tasks/scripts/release.py get-version --cargo)" >> "$GITHUB_OUTPUT" + echo "deb=$(uv run python tasks/scripts/release.py get-version --deb)" >> "$GITHUB_OUTPUT" echo "semver=${RELEASE_TAG#v}" >> "$GITHUB_OUTPUT" build-gateway: @@ -69,6 +71,13 @@ jobs: component: gateway cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + build-supervisor: + needs: [compute-versions] + uses: ./.github/workflows/docker-build.yml + with: + component: supervisor + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + build-cluster: needs: [compute-versions] uses: ./.github/workflows/docker-build.yml @@ -81,12 +90,12 @@ jobs: uses: ./.github/workflows/e2e-test.yml with: image-tag: ${{ github.sha }} - runner: build-arm64 + runner: linux-arm64-cpu8 tag-ghcr-release: name: Tag GHCR Images for Release - needs: [compute-versions, build-gateway, build-cluster, e2e] - runs-on: build-amd64 + needs: [compute-versions, build-gateway, build-supervisor, build-cluster, e2e] + runs-on: linux-amd64-cpu8 timeout-minutes: 10 steps: - name: Log in to GHCR @@ -97,7 +106,7 @@ jobs: set -euo pipefail REGISTRY="ghcr.io/nvidia/openshell" VERSION="${{ needs.compute-versions.outputs.semver }}" - for component in gateway cluster; do + for component in gateway supervisor cluster; do echo "Tagging ${REGISTRY}/${component}:${{ github.sha }} as ${VERSION} and latest..." docker buildx imagetools create \ --prefer-index=false \ @@ -116,12 +125,12 @@ jobs: matrix: include: - arch: amd64 - runner: build-amd64 + runner: linux-amd64-cpu8 artifact: linux-amd64 task: python:build:linux:amd64 output_path: target/wheels/linux-amd64/*.whl - arch: arm64 - runner: build-arm64 + runner: linux-arm64-cpu8 artifact: linux-arm64 task: python:build:linux:arm64 output_path: target/wheels/linux-arm64/*.whl @@ -135,10 +144,9 @@ jobs: options: --privileged env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} OPENSHELL_IMAGE_TAG: ${{ needs.compute-versions.outputs.semver }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -172,7 +180,7 @@ jobs: build-python-wheel-macos: name: Build Python Wheel (macOS) needs: [compute-versions] - runs-on: build-amd64 + runs-on: linux-amd64-cpu8 timeout-minutes: 120 container: image: ghcr.io/nvidia/openshell/ci:latest @@ -184,10 +192,9 @@ jobs: - /var/run/docker.sock:/var/run/docker.sock env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} OPENSHELL_IMAGE_TAG: ${{ needs.compute-versions.outputs.semver }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -197,6 +204,8 @@ jobs: - name: Set up Docker Buildx uses: ./.github/actions/setup-buildx + with: + driver: local - name: Mark workspace safe for git run: git config --global --add safe.directory "$GITHUB_WORKSPACE" @@ -230,11 +239,11 @@ jobs: matrix: include: - arch: amd64 - runner: build-amd64 + runner: linux-amd64-cpu8 target: x86_64-unknown-linux-musl zig_target: x86_64-linux-musl - arch: arm64 - runner: build-arm64 + runner: linux-arm64-cpu8 target: aarch64-unknown-linux-musl zig_target: aarch64-linux-musl runs-on: ${{ matrix.runner }} @@ -247,10 +256,9 @@ jobs: options: --privileged env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} OPENSHELL_IMAGE_TAG: ${{ needs.compute-versions.outputs.semver }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -262,7 +270,7 @@ jobs: run: git fetch --tags --force - name: Install tools - run: mise install + run: mise install --locked - name: Cache Rust target and registry uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 @@ -305,11 +313,6 @@ jobs: # Override z3-sys default (stdc++) so Rust links the matching runtime. echo "CXXSTDLIB=c++" >> "$GITHUB_ENV" - - name: Scope workspace to CLI crates - run: | - set -euo pipefail - sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-cli", "crates/openshell-core", "crates/openshell-bootstrap", "crates/openshell-policy", "crates/openshell-prover", "crates/openshell-providers", "crates/openshell-tui"]|' Cargo.toml - - name: Patch workspace version if: needs.compute-versions.outputs.cargo_version != '' run: | @@ -344,7 +347,7 @@ jobs: build-cli-macos: name: Build CLI (macOS) needs: [compute-versions] - runs-on: build-amd64 + runs-on: linux-amd64-cpu8 timeout-minutes: 60 container: image: ghcr.io/nvidia/openshell/ci:latest @@ -356,9 +359,8 @@ jobs: - /var/run/docker.sock:/var/run/docker.sock env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: ref: ${{ inputs.tag || github.ref }} fetch-depth: 0 @@ -374,6 +376,8 @@ jobs: - name: Set up Docker Buildx uses: ./.github/actions/setup-buildx + with: + driver: local - name: Build macOS binary via Docker run: | @@ -403,17 +407,296 @@ jobs: retention-days: 5 # --------------------------------------------------------------------------- - # Create a tagged GitHub Release with CLI binaries and wheels + # Build standalone gateway binaries (Linux GNU — native on each arch) + # --------------------------------------------------------------------------- + build-gateway-binary-linux: + name: Build Gateway Binary (Linux ${{ matrix.arch }}) + needs: [compute-versions] + strategy: + matrix: + include: + - arch: amd64 + runner: linux-amd64-cpu8 + target: x86_64-unknown-linux-gnu + - arch: arm64 + runner: linux-arm64-cpu8 + target: aarch64-unknown-linux-gnu + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + options: --privileged + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.tag || github.ref }} + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Install tools + run: mise install --locked + + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 + with: + shared-key: gateway-binary-gnu-${{ matrix.arch }} + cache-directories: .cache/sccache + cache-targets: "true" + + - name: Patch workspace version + if: needs.compute-versions.outputs.cargo_version != '' + run: | + set -euo pipefail + sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ needs.compute-versions.outputs.cargo_version }}"'"/}' Cargo.toml + + - name: Build ${{ matrix.target }} + run: | + set -euo pipefail + mise x -- cargo build --release --target ${{ matrix.target }} -p openshell-server + + - name: Verify packaged binary + run: | + set -euo pipefail + OUTPUT="$(target/${{ matrix.target }}/release/openshell-gateway --version)" + echo "$OUTPUT" + grep -q '^openshell-gateway ' <<<"$OUTPUT" + + - name: sccache stats + if: always() + run: mise x -- sccache --show-stats + + - name: Package binary + run: | + set -euo pipefail + mkdir -p artifacts + tar -czf artifacts/openshell-gateway-${{ matrix.target }}.tar.gz \ + -C target/${{ matrix.target }}/release openshell-gateway + ls -lh artifacts/ + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: gateway-binary-linux-${{ matrix.arch }} + path: artifacts/*.tar.gz + retention-days: 5 + + # --------------------------------------------------------------------------- + # Build standalone supervisor binaries (Linux GNU — native on each arch) + # --------------------------------------------------------------------------- + build-supervisor-binary-linux: + name: Build Supervisor Binary (Linux ${{ matrix.arch }}) + needs: [compute-versions] + strategy: + matrix: + include: + - arch: amd64 + runner: linux-amd64-cpu8 + target: x86_64-unknown-linux-gnu + - arch: arm64 + runner: linux-arm64-cpu8 + target: aarch64-unknown-linux-gnu + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + options: --privileged + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.tag || github.ref }} + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Install tools + run: mise install --locked + + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 + with: + shared-key: supervisor-binary-gnu-${{ matrix.arch }} + cache-directories: .cache/sccache + cache-targets: "true" + + - name: Patch workspace version + if: needs.compute-versions.outputs.cargo_version != '' + run: | + set -euo pipefail + sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ needs.compute-versions.outputs.cargo_version }}"'"/}' Cargo.toml + + - name: Build ${{ matrix.target }} + run: | + set -euo pipefail + mise x -- cargo build --release --target ${{ matrix.target }} -p openshell-sandbox --bin openshell-sandbox + + - name: Verify packaged binary + run: | + set -euo pipefail + OUTPUT="$(target/${{ matrix.target }}/release/openshell-sandbox --version)" + echo "$OUTPUT" + grep -q '^openshell-sandbox ' <<<"$OUTPUT" + + - name: sccache stats + if: always() + run: mise x -- sccache --show-stats + + - name: Package binary + run: | + set -euo pipefail + mkdir -p artifacts + tar -czf artifacts/openshell-sandbox-${{ matrix.target }}.tar.gz \ + -C target/${{ matrix.target }}/release openshell-sandbox + ls -lh artifacts/ + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: supervisor-binary-linux-${{ matrix.arch }} + path: artifacts/*.tar.gz + retention-days: 5 + + # --------------------------------------------------------------------------- + # Build standalone gateway binary (macOS aarch64 via osxcross) + # --------------------------------------------------------------------------- + build-gateway-binary-macos: + name: Build Gateway Binary (macOS) + needs: [compute-versions] + runs-on: linux-amd64-cpu8 + timeout-minutes: 60 + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + options: --privileged + volumes: + - /var/run/docker.sock:/var/run/docker.sock + env: + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.tag || github.ref }} + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Log in to GHCR + run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin + + - name: Set up Docker Buildx + uses: ./.github/actions/setup-buildx + with: + driver: local + + - name: Build macOS binary via Docker + run: | + set -euo pipefail + docker buildx build \ + --file deploy/docker/Dockerfile.gateway-macos \ + --build-arg OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \ + --build-arg CARGO_TARGET_CACHE_SCOPE="${{ github.sha }}" \ + --target binary \ + --output type=local,dest=out/ \ + . + + - name: Verify packaged binary shape + run: | + set -euo pipefail + test -x out/openshell-gateway + + - name: Package binary + run: | + set -euo pipefail + mkdir -p artifacts + tar -czf artifacts/openshell-gateway-aarch64-apple-darwin.tar.gz \ + -C out openshell-gateway + ls -lh artifacts/ + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: gateway-binary-macos + path: artifacts/*.tar.gz + retention-days: 5 + + build-driver-vm-linux: + name: Build Driver VM Linux + needs: [compute-versions] + uses: ./.github/workflows/driver-vm-linux.yml + with: + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + image-tag: ${{ needs.compute-versions.outputs.semver }} + checkout-ref: ${{ inputs.tag || github.ref }} + secrets: inherit + + build-driver-vm-macos: + name: Build Driver VM macOS + needs: [compute-versions] + uses: ./.github/workflows/driver-vm-macos.yml + with: + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + image-tag: ${{ needs.compute-versions.outputs.semver }} + checkout-ref: ${{ inputs.tag || github.ref }} + secrets: inherit + + build-deb: + name: Build Debian Packages + needs: [compute-versions, build-cli-linux, build-gateway-binary-linux, build-driver-vm-linux] + uses: ./.github/workflows/deb-package.yml + with: + deb-version: ${{ needs.compute-versions.outputs.deb_version }} + checkout-ref: ${{ inputs.tag || github.ref }} + secrets: inherit + + build-rpm: + name: Build RPM Packages + needs: [compute-versions] + uses: ./.github/workflows/rpm-package.yml + with: + checkout-ref: ${{ inputs.tag || github.ref }} + secrets: inherit + + # --------------------------------------------------------------------------- + # Create a tagged GitHub Release with CLI, gateway, driver, and wheels # --------------------------------------------------------------------------- release: name: Release - needs: [compute-versions, build-cli-linux, build-cli-macos, build-python-wheels-linux, build-python-wheel-macos, tag-ghcr-release] - runs-on: build-amd64 + needs: [compute-versions, build-cli-linux, build-cli-macos, build-gateway-binary-linux, build-gateway-binary-macos, build-supervisor-binary-linux, build-python-wheels-linux, build-python-wheel-macos, tag-ghcr-release, build-driver-vm-linux, build-driver-vm-macos, build-deb, build-rpm] + runs-on: linux-amd64-cpu8 timeout-minutes: 10 + permissions: + contents: write + id-token: write + attestations: write + artifact-metadata: write outputs: wheel_filenames: ${{ steps.wheel_filenames.outputs.wheel_filenames }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: ref: ${{ inputs.tag || github.ref }} @@ -424,6 +707,27 @@ jobs: path: release/ merge-multiple: true + - name: Download gateway binary artifacts + uses: actions/download-artifact@v4 + with: + pattern: gateway-binary-* + path: release/ + merge-multiple: true + + - name: Download supervisor binary artifacts + uses: actions/download-artifact@v4 + with: + pattern: supervisor-binary-* + path: release/ + merge-multiple: true + + - name: Download VM driver artifacts + uses: actions/download-artifact@v4 + with: + pattern: driver-vm-* + path: release/ + merge-multiple: true + - name: Download wheel artifacts uses: actions/download-artifact@v4 with: @@ -431,6 +735,20 @@ jobs: path: release/ merge-multiple: true + - name: Download Debian package artifacts + uses: actions/download-artifact@v4 + with: + pattern: deb-linux-* + path: release/ + merge-multiple: true + + - name: Download RPM package artifacts + uses: actions/download-artifact@v4 + with: + pattern: rpm-linux-* + path: release/ + merge-multiple: true + - name: Capture wheel filenames id: wheel_filenames run: | @@ -443,8 +761,53 @@ jobs: run: | set -euo pipefail cd release - sha256sum *.tar.gz *.whl > openshell-checksums-sha256.txt + sha256sum \ + openshell-x86_64-unknown-linux-musl.tar.gz \ + openshell-aarch64-unknown-linux-musl.tar.gz \ + openshell-aarch64-apple-darwin.tar.gz \ + openshell_*.deb \ + openshell-*.rpm \ + *.whl > openshell-checksums-sha256.txt cat openshell-checksums-sha256.txt + sha256sum \ + openshell-gateway-x86_64-unknown-linux-gnu.tar.gz \ + openshell-gateway-aarch64-unknown-linux-gnu.tar.gz \ + openshell-gateway-aarch64-apple-darwin.tar.gz > openshell-gateway-checksums-sha256.txt + cat openshell-gateway-checksums-sha256.txt + sha256sum \ + openshell-sandbox-x86_64-unknown-linux-gnu.tar.gz \ + openshell-sandbox-aarch64-unknown-linux-gnu.tar.gz > openshell-sandbox-checksums-sha256.txt + cat openshell-sandbox-checksums-sha256.txt + + - name: Attest VM driver artifacts + uses: actions/attest@v4 + with: + subject-path: | + release/openshell-driver-vm-x86_64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-aarch64-apple-darwin.tar.gz + + - name: Prune removed VM checksum asset + uses: actions/github-script@v7 + with: + script: | + const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/'); + let release; + try { + release = await github.rest.repos.getReleaseByTag({ owner, repo, tag: process.env.RELEASE_TAG }); + } catch (err) { + if (err.status === 404) { + core.info(`No existing ${process.env.RELEASE_TAG} release; skipping VM checksum pruning.`); + return; + } + throw err; + } + for (const asset of release.data.assets) { + if (asset.name === 'openshell-driver-vm-checksums-sha256.txt') { + core.info(`Deleting removed VM checksum asset: ${asset.name}`); + await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: asset.id }); + } + } - name: Create GitHub Release uses: softprops/action-gh-release@v2 @@ -466,8 +829,20 @@ jobs: release/openshell-x86_64-unknown-linux-musl.tar.gz release/openshell-aarch64-unknown-linux-musl.tar.gz release/openshell-aarch64-apple-darwin.tar.gz + release/openshell_*.deb + release/openshell-*.rpm + release/openshell-gateway-x86_64-unknown-linux-gnu.tar.gz + release/openshell-gateway-aarch64-unknown-linux-gnu.tar.gz + release/openshell-gateway-aarch64-apple-darwin.tar.gz + release/openshell-sandbox-x86_64-unknown-linux-gnu.tar.gz + release/openshell-sandbox-aarch64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-x86_64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-aarch64-unknown-linux-gnu.tar.gz + release/openshell-driver-vm-aarch64-apple-darwin.tar.gz release/*.whl release/openshell-checksums-sha256.txt + release/openshell-gateway-checksums-sha256.txt + release/openshell-sandbox-checksums-sha256.txt publish-fern-docs: name: Publish Fern Docs @@ -475,7 +850,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 15 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 with: ref: ${{ inputs.tag || github.ref }} diff --git a/.github/workflows/release-vm-dev.yml b/.github/workflows/release-vm-dev.yml deleted file mode 100644 index 06ba474a07..0000000000 --- a/.github/workflows/release-vm-dev.yml +++ /dev/null @@ -1,535 +0,0 @@ -name: Release VM Dev - -# Build openshell-vm binaries for all supported platforms and upload them to -# the rolling "vm-dev" GitHub Release. Each binary is self-extracting: it -# embeds pre-built kernel runtime artifacts (from release-vm-kernel.yml) and a -# base rootfs tarball. -# -# Prerequisites: the vm-dev release must already contain kernel runtime -# tarballs. Run the "Release VM Kernel" workflow first if they are missing. - -on: - push: - branches: [main] - workflow_dispatch: - -permissions: - contents: write - packages: read - -# Serialize with release-vm-kernel.yml — both update the vm-dev release. -concurrency: - group: vm-dev-release - cancel-in-progress: false - -defaults: - run: - shell: bash - -jobs: - # --------------------------------------------------------------------------- - # Compute versions (reuse the same logic as release-dev.yml) - # --------------------------------------------------------------------------- - compute-versions: - name: Compute Versions - runs-on: build-amd64 - timeout-minutes: 5 - container: - image: ghcr.io/nvidia/openshell/ci:latest - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - outputs: - cargo_version: ${{ steps.v.outputs.cargo }} - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" - - - name: Fetch tags - run: git fetch --tags --force - - - name: Compute versions - id: v - run: | - set -euo pipefail - echo "cargo=$(uv run python tasks/scripts/release.py get-version --cargo)" >> "$GITHUB_OUTPUT" - - # --------------------------------------------------------------------------- - # Download kernel runtime tarballs from the vm-dev release - # --------------------------------------------------------------------------- - download-kernel-runtime: - name: Download Kernel Runtime - runs-on: build-amd64 - timeout-minutes: 10 - container: - image: ghcr.io/nvidia/openshell/ci:latest - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - steps: - - uses: actions/checkout@v4 - - - name: Download all runtime tarballs - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - set -euo pipefail - mkdir -p runtime-artifacts - - for platform in linux-aarch64 linux-x86_64 darwin-aarch64; do - echo "Downloading vm-runtime-${platform}.tar.zst..." - gh release download vm-dev \ - --repo "${GITHUB_REPOSITORY}" \ - --pattern "vm-runtime-${platform}.tar.zst" \ - --dir runtime-artifacts \ - --clobber - done - - echo "Downloaded runtime artifacts:" - ls -lah runtime-artifacts/ - - - name: Verify downloads - run: | - set -euo pipefail - for platform in linux-aarch64 linux-x86_64 darwin-aarch64; do - file="runtime-artifacts/vm-runtime-${platform}.tar.zst" - if [ ! -f "$file" ]; then - echo "ERROR: Missing ${file}" >&2 - echo "" >&2 - echo "The vm-dev release does not have kernel runtime artifacts." >&2 - echo "Run the 'Release VM Kernel' workflow first:" >&2 - echo " gh workflow run release-vm-kernel.yml" >&2 - exit 1 - fi - echo "OK: ${file} ($(du -sh "$file" | cut -f1))" - done - - - name: Upload as workflow artifact - uses: actions/upload-artifact@v4 - with: - name: kernel-runtime-tarballs - path: runtime-artifacts/vm-runtime-*.tar.zst - retention-days: 1 - - # --------------------------------------------------------------------------- - # Build base rootfs tarballs (architecture-specific) - # --------------------------------------------------------------------------- - build-rootfs: - name: Build Rootfs (${{ matrix.arch }}) - needs: [compute-versions] - strategy: - matrix: - include: - - arch: arm64 - runner: build-arm64 - guest_arch: aarch64 - - arch: amd64 - runner: build-amd64 - guest_arch: x86_64 - runs-on: ${{ matrix.runner }} - timeout-minutes: 30 - container: - image: ghcr.io/nvidia/openshell/ci:latest - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - options: --privileged - volumes: - - /var/run/docker.sock:/var/run/docker.sock - env: - MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - OPENSHELL_IMAGE_TAG: dev - steps: - - uses: actions/checkout@v4 - - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" - - - name: Log in to GHCR - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin - - - name: Install tools - run: mise install - - - name: Install zstd - run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/* - - - name: Build base rootfs tarball - run: | - set -euo pipefail - crates/openshell-vm/scripts/build-rootfs.sh \ - --base \ - --arch ${{ matrix.guest_arch }} \ - target/rootfs-build - - mkdir -p target/vm-runtime-compressed - tar -C target/rootfs-build -cf - . \ - | zstd -19 -T0 -o target/vm-runtime-compressed/rootfs.tar.zst - - echo "Rootfs tarball: $(du -sh target/vm-runtime-compressed/rootfs.tar.zst | cut -f1)" - - - name: Upload rootfs artifact - uses: actions/upload-artifact@v4 - with: - name: rootfs-${{ matrix.arch }} - path: target/vm-runtime-compressed/rootfs.tar.zst - retention-days: 1 - - # --------------------------------------------------------------------------- - # Build openshell-vm binary (Linux — native on each arch) - # --------------------------------------------------------------------------- - build-vm-linux: - name: Build VM (Linux ${{ matrix.arch }}) - needs: [compute-versions, download-kernel-runtime, build-rootfs] - strategy: - matrix: - include: - - arch: arm64 - runner: build-arm64 - target: aarch64-unknown-linux-gnu - platform: linux-aarch64 - guest_arch: aarch64 - - arch: amd64 - runner: build-amd64 - target: x86_64-unknown-linux-gnu - platform: linux-x86_64 - guest_arch: x86_64 - runs-on: ${{ matrix.runner }} - timeout-minutes: 30 - container: - image: ghcr.io/nvidia/openshell/ci:latest - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - options: --privileged - env: - MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} - OPENSHELL_IMAGE_TAG: dev - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" - - - name: Fetch tags - run: git fetch --tags --force - - - name: Install tools - run: mise install - - - name: Cache Rust target and registry - uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 - with: - shared-key: vm-linux-${{ matrix.arch }} - cache-directories: .cache/sccache - cache-targets: "true" - - - name: Install zstd - run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/* - - - name: Download kernel runtime tarball - uses: actions/download-artifact@v4 - with: - name: kernel-runtime-tarballs - path: runtime-download/ - - - name: Download rootfs tarball - uses: actions/download-artifact@v4 - with: - name: rootfs-${{ matrix.arch }} - path: rootfs-download/ - - - name: Stage compressed runtime for embedding - run: | - set -euo pipefail - COMPRESSED_DIR="${PWD}/target/vm-runtime-compressed" - mkdir -p "$COMPRESSED_DIR" - - # Extract kernel runtime tarball and re-compress individual files - EXTRACT_DIR=$(mktemp -d) - zstd -d "runtime-download/vm-runtime-${{ matrix.platform }}.tar.zst" --stdout \ - | tar -xf - -C "$EXTRACT_DIR" - - echo "Extracted runtime files:" - ls -lah "$EXTRACT_DIR" - - for file in "$EXTRACT_DIR"/*; do - [ -f "$file" ] || continue - name=$(basename "$file") - [ "$name" = "provenance.json" ] && continue - zstd -19 -f -q -T0 -o "${COMPRESSED_DIR}/${name}.zst" "$file" - done - - # Copy rootfs tarball (already zstd-compressed) - cp rootfs-download/rootfs.tar.zst "${COMPRESSED_DIR}/rootfs.tar.zst" - - echo "Staged compressed artifacts:" - ls -lah "$COMPRESSED_DIR" - - - name: Scope workspace to VM crates - run: | - set -euo pipefail - sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-vm", "crates/openshell-core", "crates/openshell-bootstrap", "crates/openshell-policy"]|' Cargo.toml - - - name: Patch workspace version - if: needs.compute-versions.outputs.cargo_version != '' - run: | - set -euo pipefail - sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ needs.compute-versions.outputs.cargo_version }}"'"/}' Cargo.toml - - - name: Build openshell-vm - run: | - set -euo pipefail - OPENSHELL_VM_RUNTIME_COMPRESSED_DIR="${PWD}/target/vm-runtime-compressed" \ - mise x -- cargo build --release -p openshell-vm - - - name: sccache stats - if: always() - run: mise x -- sccache --show-stats - - - name: Package binary - run: | - set -euo pipefail - mkdir -p artifacts - tar -czf "artifacts/openshell-vm-${{ matrix.target }}.tar.gz" \ - -C target/release openshell-vm - ls -lh artifacts/ - - - name: Upload artifact - uses: actions/upload-artifact@v4 - with: - name: vm-linux-${{ matrix.arch }} - path: artifacts/*.tar.gz - retention-days: 5 - - # --------------------------------------------------------------------------- - # Build openshell-vm binary (macOS ARM64 via osxcross) - # --------------------------------------------------------------------------- - build-vm-macos: - name: Build VM (macOS) - needs: [compute-versions, download-kernel-runtime, build-rootfs] - runs-on: build-amd64 - timeout-minutes: 60 - container: - image: ghcr.io/nvidia/openshell/ci:latest - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - options: --privileged - volumes: - - /var/run/docker.sock:/var/run/docker.sock - env: - MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_MEMCACHED_ENDPOINT: ${{ vars.SCCACHE_MEMCACHED_ENDPOINT }} - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" - - - name: Fetch tags - run: git fetch --tags --force - - - name: Log in to GHCR - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin - - - name: Set up Docker Buildx - uses: ./.github/actions/setup-buildx - - - name: Install zstd - run: apt-get update && apt-get install -y --no-install-recommends zstd && rm -rf /var/lib/apt/lists/* - - - name: Download kernel runtime tarball - uses: actions/download-artifact@v4 - with: - name: kernel-runtime-tarballs - path: runtime-download/ - - - name: Download rootfs tarball (arm64) - uses: actions/download-artifact@v4 - with: - name: rootfs-arm64 - path: rootfs-download/ - - - name: Prepare compressed runtime directory - run: | - set -euo pipefail - COMPRESSED_DIR="${PWD}/target/vm-runtime-compressed-macos" - mkdir -p "$COMPRESSED_DIR" - - # Extract the darwin runtime tarball and re-compress for embedding. - # The macOS embedded.rs expects: libkrun.dylib.zst, libkrunfw.5.dylib.zst, gvproxy.zst - EXTRACT_DIR=$(mktemp -d) - zstd -d "runtime-download/vm-runtime-darwin-aarch64.tar.zst" --stdout \ - | tar -xf - -C "$EXTRACT_DIR" - - echo "Extracted darwin runtime files:" - ls -lah "$EXTRACT_DIR" - - for file in "$EXTRACT_DIR"/*; do - [ -f "$file" ] || continue - name=$(basename "$file") - [ "$name" = "provenance.json" ] && continue - zstd -19 -f -q -T0 -o "${COMPRESSED_DIR}/${name}.zst" "$file" - done - - # The macOS VM guest is always Linux ARM64, so use the arm64 rootfs - cp rootfs-download/rootfs.tar.zst "${COMPRESSED_DIR}/rootfs.tar.zst" - - echo "Staged macOS compressed artifacts:" - ls -lah "$COMPRESSED_DIR" - - - name: Build macOS binary via Docker (osxcross) - run: | - set -euo pipefail - docker buildx build \ - --file deploy/docker/Dockerfile.vm-macos \ - --build-arg OPENSHELL_CARGO_VERSION="${{ needs.compute-versions.outputs.cargo_version }}" \ - --build-arg OPENSHELL_IMAGE_TAG=dev \ - --build-arg CARGO_TARGET_CACHE_SCOPE="${{ github.sha }}" \ - --build-context vm-runtime-compressed="${PWD}/target/vm-runtime-compressed-macos" \ - --target binary \ - --output type=local,dest=out/ \ - . - - - name: Package binary - run: | - set -euo pipefail - mkdir -p artifacts - tar -czf artifacts/openshell-vm-aarch64-apple-darwin.tar.gz \ - -C out openshell-vm - ls -lh artifacts/ - - - name: Upload artifact - uses: actions/upload-artifact@v4 - with: - name: vm-macos - path: artifacts/*.tar.gz - retention-days: 5 - - # --------------------------------------------------------------------------- - # Upload all VM binaries to the vm-dev rolling release - # --------------------------------------------------------------------------- - release-vm-dev: - name: Release VM Dev - needs: [build-vm-linux, build-vm-macos] - runs-on: build-amd64 - timeout-minutes: 10 - steps: - - uses: actions/checkout@v4 - - - name: Download all VM binary artifacts - uses: actions/download-artifact@v4 - with: - pattern: vm-* - path: release/ - merge-multiple: true - - - name: Filter to only binary tarballs - run: | - set -euo pipefail - mkdir -p release-final - # Only include the openshell-vm binary tarballs, not kernel runtime - cp release/openshell-vm-*.tar.gz release-final/ - count=$(ls release-final/openshell-vm-*.tar.gz 2>/dev/null | wc -l) - if [ "$count" -eq 0 ]; then - echo "ERROR: No VM binary tarballs found in release/" >&2 - exit 1 - fi - echo "Release artifacts (${count} binaries):" - ls -lh release-final/ - - - name: Generate checksums - run: | - set -euo pipefail - cd release-final - sha256sum openshell-vm-*.tar.gz > vm-binary-checksums-sha256.txt - cat vm-binary-checksums-sha256.txt - - - name: Ensure vm-dev tag exists - run: | - git config user.name "github-actions[bot]" - git config user.email "github-actions[bot]@users.noreply.github.com" - git tag -fa vm-dev -m "VM Development Build" "${GITHUB_SHA}" - git push --force origin vm-dev - - - name: Prune stale VM binary assets from vm-dev release - uses: actions/github-script@v7 - with: - script: | - const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/'); - let release; - try { - release = await github.rest.repos.getReleaseByTag({ owner, repo, tag: 'vm-dev' }); - } catch (err) { - if (err.status === 404) { - core.info('No existing vm-dev release; will create fresh.'); - return; - } - throw err; - } - // Delete old VM binary assets (keep kernel runtime assets) - for (const asset of release.data.assets) { - if (asset.name.startsWith('openshell-vm-') || asset.name === 'vm-binary-checksums-sha256.txt') { - core.info(`Deleting stale asset: ${asset.name}`); - await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: asset.id }); - } - } - - - name: Upload to vm-dev GitHub Release - uses: softprops/action-gh-release@v2 - with: - name: OpenShell VM Development Build - prerelease: true - tag_name: vm-dev - target_commitish: ${{ github.sha }} - body: | - Rolling development build of **openshell-vm** — the MicroVM runtime for OpenShell. - - > **NOTE**: This is a development build, not a tagged release, and may be unstable. - - ### Kernel Runtime Artifacts - - Pre-built kernel runtime (libkrunfw + libkrun + gvproxy) for embedding into - the openshell-vm binary. These are rebuilt when the kernel config or pinned - dependency versions change. - - | Platform | Artifact | - |----------|----------| - | Linux ARM64 | `vm-runtime-linux-aarch64.tar.zst` | - | Linux x86_64 | `vm-runtime-linux-x86_64.tar.zst` | - | macOS ARM64 | `vm-runtime-darwin-aarch64.tar.zst` | - - ### VM Binaries - - Self-extracting openshell-vm binaries with embedded kernel runtime and base - rootfs. These are rebuilt on every push to main. - - | Platform | Artifact | - |----------|----------| - | Linux ARM64 | `openshell-vm-aarch64-unknown-linux-gnu.tar.gz` | - | Linux x86_64 | `openshell-vm-x86_64-unknown-linux-gnu.tar.gz` | - | macOS ARM64 | `openshell-vm-aarch64-apple-darwin.tar.gz` | - - ### Quick install - - ``` - curl -fsSL https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install-vm.sh | sh - ``` - - Auto-detects your platform, verifies checksums, and codesigns on macOS. - - files: | - release-final/openshell-vm-aarch64-unknown-linux-gnu.tar.gz - release-final/openshell-vm-x86_64-unknown-linux-gnu.tar.gz - release-final/openshell-vm-aarch64-apple-darwin.tar.gz - release-final/vm-binary-checksums-sha256.txt diff --git a/.github/workflows/release-vm-kernel.yml b/.github/workflows/release-vm-kernel.yml index c1593da313..8bdaab11f7 100644 --- a/.github/workflows/release-vm-kernel.yml +++ b/.github/workflows/release-vm-kernel.yml @@ -1,16 +1,16 @@ name: Release VM Kernel # Build custom libkrunfw (kernel firmware) + libkrun (VMM) + gvproxy for all -# supported openshell-vm platforms. Artifacts are uploaded to the rolling -# "vm-dev" GitHub Release and consumed by release-vm-dev.yml when building the -# openshell-vm binary. +# supported openshell-driver-vm platforms. Artifacts are uploaded to the +# rolling "vm-runtime" GitHub Release and consumed by normal dev/tag release +# workflows when building the openshell-driver-vm binary. # # The Linux kernel is compiled once on aarch64 Linux. The resulting kernel.c # (a C source file containing the kernel as a byte array) is shared with the # macOS job, which only needs to compile it into a .dylib — no krunvm, no # Fedora VM, no kernel rebuild. This cuts macOS CI from ~45 min to ~5 min. # -# This workflow runs on-demand (or when kernel config / pins change). It is +# This workflow runs on demand when kernel config or pins change. It is # intentionally decoupled from the per-commit VM binary build because the # kernel rarely changes and takes 15-45 minutes to compile. @@ -21,9 +21,9 @@ permissions: contents: write packages: read -# Serialize with release-vm-dev.yml — both update the vm-dev release. +# Serialize runtime release updates. concurrency: - group: vm-dev-release + group: vm-runtime-release cancel-in-progress: false defaults: @@ -36,7 +36,7 @@ jobs: # --------------------------------------------------------------------------- build-runtime-linux-arm64: name: Build Runtime (Linux ARM64) - runs-on: build-arm64 + runs-on: linux-arm64-cpu8 timeout-minutes: 60 container: image: ghcr.io/nvidia/openshell/ci:latest @@ -47,7 +47,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Mark workspace safe for git run: git config --global --add safe.directory "$GITHUB_WORKSPACE" @@ -86,7 +86,7 @@ jobs: # --------------------------------------------------------------------------- build-runtime-linux-amd64: name: Build Runtime (Linux AMD64) - runs-on: build-amd64 + runs-on: linux-amd64-cpu8 timeout-minutes: 60 container: image: ghcr.io/nvidia/openshell/ci:latest @@ -97,7 +97,7 @@ jobs: env: MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Mark workspace safe for git run: git config --global --add safe.directory "$GITHUB_WORKSPACE" @@ -130,12 +130,12 @@ jobs: env: RUSTC_WRAPPER: "" steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Install dependencies run: | set -euo pipefail - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.95.0 echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" brew install lld dtc xz @@ -163,15 +163,20 @@ jobs: retention-days: 5 # --------------------------------------------------------------------------- - # Upload all runtime tarballs to the vm-dev rolling release + # Upload all runtime tarballs to the vm-runtime rolling release # --------------------------------------------------------------------------- release-kernel: name: Release Kernel Runtime needs: [build-runtime-linux-arm64, build-runtime-linux-amd64, build-runtime-macos-arm64] - runs-on: build-amd64 + runs-on: linux-amd64-cpu8 timeout-minutes: 10 + permissions: + contents: write + id-token: write + attestations: write + artifact-metadata: write steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Download all runtime artifacts uses: actions/download-artifact@v4 @@ -180,36 +185,37 @@ jobs: path: release/ merge-multiple: true - - name: Generate checksums - run: | - set -euo pipefail - cd release - sha256sum vm-runtime-*.tar.zst > vm-runtime-checksums-sha256.txt - cat vm-runtime-checksums-sha256.txt + - name: Attest VM runtime artifacts + uses: actions/attest@v4 + with: + subject-path: | + release/vm-runtime-linux-aarch64.tar.zst + release/vm-runtime-linux-x86_64.tar.zst + release/vm-runtime-darwin-aarch64.tar.zst - - name: Ensure vm-dev tag exists + - name: Ensure vm-runtime tag exists run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" - git tag -fa vm-dev -m "VM Development Build" "${GITHUB_SHA}" - git push --force origin vm-dev + git tag -fa vm-runtime -m "VM Runtime Development Build" "${GITHUB_SHA}" + git push --force origin vm-runtime - - name: Prune stale runtime assets from vm-dev release + - name: Prune stale runtime assets from vm-runtime release uses: actions/github-script@v7 with: script: | const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/'); let release; try { - release = await github.rest.repos.getReleaseByTag({ owner, repo, tag: 'vm-dev' }); + release = await github.rest.repos.getReleaseByTag({ owner, repo, tag: 'vm-runtime' }); } catch (err) { if (err.status === 404) { - core.info('No existing vm-dev release; will create fresh.'); + core.info('No existing vm-runtime release; will create fresh.'); return; } throw err; } - // Delete old runtime tarballs and checksums (keep vm binary assets) + // Delete old runtime assets, including removed checksum files. for (const asset of release.data.assets) { if (asset.name.startsWith('vm-runtime-')) { core.info(`Deleting stale asset: ${asset.name}`); @@ -217,25 +223,23 @@ jobs: } } - - name: Create / update vm-dev GitHub Release + - name: Create / update vm-runtime GitHub Release uses: softprops/action-gh-release@v2 with: - name: OpenShell VM Development Build + name: OpenShell VM Runtime prerelease: true - tag_name: vm-dev + tag_name: vm-runtime target_commitish: ${{ github.sha }} body: | - Rolling development build of **openshell-vm** — the MicroVM runtime for OpenShell. + Build of the OpenShell VM runtime artifacts used by `openshell-driver-vm`. - > **NOTE**: This is a development build, not a tagged release, and may be unstable. - > The VM implementation itself is also experimental and may change or break without - > notice. + > **NOTE**: This is a development build. ### Kernel Runtime Artifacts Pre-built kernel runtime (libkrunfw + libkrun + gvproxy) for embedding into - the openshell-vm binary. These are rebuilt when the kernel config or pinned - dependency versions change. + the `openshell-driver-vm` binary. These are rebuilt on demand when the kernel + config or pinned dependency versions change. | Platform | Artifact | |----------|----------| @@ -243,27 +247,14 @@ jobs: | Linux x86_64 | `vm-runtime-linux-x86_64.tar.zst` | | macOS ARM64 | `vm-runtime-darwin-aarch64.tar.zst` | - ### VM Binaries - - Self-extracting openshell-vm binaries with embedded kernel runtime and base - rootfs. These are rebuilt on every push to main. + ### Verify - | Platform | Artifact | - |----------|----------| - | Linux ARM64 | `openshell-vm-aarch64-unknown-linux-gnu.tar.gz` | - | Linux x86_64 | `openshell-vm-x86_64-unknown-linux-gnu.tar.gz` | - | macOS ARM64 | `openshell-vm-aarch64-apple-darwin.tar.gz` | - - ### Quick install - - ``` - curl -fsSL https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install-vm.sh | sh + ```bash + gh release download vm-runtime -R NVIDIA/OpenShell -p vm-runtime-linux-x86_64.tar.zst + gh attestation verify vm-runtime-linux-x86_64.tar.zst -R NVIDIA/OpenShell ``` - Auto-detects your platform, verifies checksums, and codesigns on macOS. - files: | release/vm-runtime-linux-aarch64.tar.zst release/vm-runtime-linux-x86_64.tar.zst release/vm-runtime-darwin-aarch64.tar.zst - release/vm-runtime-checksums-sha256.txt diff --git a/.github/workflows/rpm-package.yml b/.github/workflows/rpm-package.yml new file mode 100644 index 0000000000..f6003d6669 --- /dev/null +++ b/.github/workflows/rpm-package.yml @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: RPM Package + +on: + workflow_call: + inputs: + checkout-ref: + required: true + type: string + +permissions: + contents: read + +defaults: + run: + shell: bash + +jobs: + build-rpm-linux: + name: Build RPM Package (Linux ${{ matrix.arch }}) + strategy: + matrix: + include: + - arch: x86_64 + runner: linux-amd64-cpu8 + - arch: aarch64 + runner: linux-arm64-cpu8 + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + container: + image: fedora:latest + steps: + - name: Install build dependencies + run: | + dnf install -y \ + packit rpm-build \ + rust cargo gcc gcc-c++ make cmake pkg-config \ + clang-devel z3-devel systemd-rpm-macros \ + pandoc python3-devel git-core \ + cargo-rpm-macros + + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.checkout-ref }} + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Build RPMs via Packit + run: packit build locally + + - name: Collect RPM artifacts + run: | + set -euo pipefail + mkdir -p artifacts + for rpm_dir in "$HOME/rpmbuild/RPMS" "$PWD/${{ matrix.arch }}"; do + if [ -d "$rpm_dir" ]; then + find "$rpm_dir" -type f -name '*.rpm' -exec cp {} artifacts/ \; + fi + done + if ! compgen -G 'artifacts/*.rpm' > /dev/null; then + echo "::error::No RPM artifacts found" + find "$PWD" -maxdepth 3 -type f -name '*.rpm' -print + exit 1 + fi + echo "=== Built RPMs ===" + ls -lah artifacts/ + + - name: Upload RPM artifacts + uses: actions/upload-artifact@v4 + with: + name: rpm-linux-${{ matrix.arch }} + path: artifacts/*.rpm + retention-days: 5 diff --git a/.github/workflows/shadow-docker-build.yml b/.github/workflows/shadow-docker-build.yml new file mode 100644 index 0000000000..62e6878675 --- /dev/null +++ b/.github/workflows/shadow-docker-build.yml @@ -0,0 +1,43 @@ +name: Shadow Docker Build + +# OS-128 Phase 4: manual non-publishing exercise of the production Docker +# image workflow. This stays off main's push surface because the image path is +# not a required signal while the prebuilt-binary rollout is being measured. + +on: + workflow_dispatch: + inputs: + platform: + description: "Target platform(s)" + required: false + type: string + default: "linux/amd64,linux/arm64" + +permissions: + contents: read + packages: write + +jobs: + gateway: + uses: ./.github/workflows/docker-build.yml + with: + component: gateway + platform: ${{ inputs.platform }} + push: false + secrets: inherit + + supervisor: + uses: ./.github/workflows/docker-build.yml + with: + component: supervisor + platform: ${{ inputs.platform }} + push: false + secrets: inherit + + cluster: + uses: ./.github/workflows/docker-build.yml + with: + component: cluster + platform: ${{ inputs.platform }} + push: false + secrets: inherit diff --git a/.github/workflows/shadow-rust-native-build.yml b/.github/workflows/shadow-rust-native-build.yml new file mode 100644 index 0000000000..80df1056f7 --- /dev/null +++ b/.github/workflows/shadow-rust-native-build.yml @@ -0,0 +1,255 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Rust Native Build (openshell-gateway / openshell-sandbox) + +# OS-128 Phase 4: build Rust binaries natively per Linux architecture before +# the Docker image build consumes them as prebuilt artifacts. + +on: + workflow_call: + inputs: + component: + description: "Binary component to build (gateway or sandbox)" + required: true + type: string + arch: + description: "Linux architecture to build (amd64 or arm64)" + required: true + type: string + cargo-version: + description: "Pre-computed cargo version (skips internal git-based computation)" + required: false + type: string + default: "" + features: + description: "Cargo features to enable" + required: false + type: string + default: "openshell-core/dev-settings" + retention-days: + description: "Artifact retention period" + required: false + type: number + default: 5 + artifact-name: + description: "Artifact name override" + required: false + type: string + default: "" + workflow_dispatch: + inputs: + component: + description: "Binary component to build" + required: true + type: choice + default: gateway + options: + - gateway + - sandbox + arch: + description: "Linux architecture to build" + required: true + type: choice + default: amd64 + options: + - amd64 + - arm64 + cargo-version: + description: "Cargo version override" + required: false + type: string + default: "" + features: + description: "Cargo features to enable" + required: false + type: string + default: "openshell-core/dev-settings" + retention-days: + description: "Artifact retention period" + required: false + type: number + default: 5 + artifact-name: + description: "Artifact name override" + required: false + type: string + default: "" + +permissions: + contents: read + packages: read + +env: + CARGO_TERM_COLOR: always + CARGO_INCREMENTAL: "0" + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Route sccache (already RUSTC_WRAPPER in mise.toml) to the GHA cache + # backend instead of the EKS memcached used by ARC. + SCCACHE_GHA_ENABLED: "true" + +defaults: + run: + shell: bash + +jobs: + rust-native-build: + name: ${{ inputs.component }} (${{ inputs.arch }}) + runs-on: ${{ inputs.arch == 'arm64' && 'linux-arm64-cpu8' || 'linux-amd64-cpu8' }} + timeout-minutes: 60 + env: + COMPONENT: ${{ inputs.component }} + ARCH: ${{ inputs.arch }} + FEATURES: ${{ inputs.features }} + # Partition the GHA sccache cache per (component, arch). Without this, + # concurrent jobs collide on the same cache key and later-starting + # writers hit 409 Conflict (PR #961 fix for shadow-shared-cpu-spike). + SCCACHE_GHA_VERSION: ${{ inputs.component }}-${{ inputs.arch }} + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Install tools + run: mise install --locked + + - name: Resolve build target + id: target + run: | + set -euo pipefail + + case "$COMPONENT" in + gateway) + crate=openshell-server + binary=openshell-gateway + ;; + sandbox) + crate=openshell-sandbox + binary=openshell-sandbox + ;; + *) + echo "unsupported component: $COMPONENT" >&2 + exit 1 + ;; + esac + + case "$ARCH" in + amd64) + target=x86_64-unknown-linux-gnu + ;; + arm64) + target=aarch64-unknown-linux-gnu + ;; + *) + echo "unsupported arch: $ARCH" >&2 + exit 1 + ;; + esac + + { + echo "crate=$crate" + echo "binary=$binary" + echo "target=$target" + } >> "$GITHUB_OUTPUT" + + - name: Configure GHA sccache backend + # Exposes ACTIONS_CACHE_URL / ACTIONS_RUNTIME_TOKEN so sccache (wrapped + # around rustc via mise's RUSTC_WRAPPER) can initialize the GHA cache. + uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9 + + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 + with: + shared-key: rust-native-${{ inputs.component }}-${{ inputs.arch }} + cache-directories: .cache/sccache + cache-targets: "true" + + - name: Compute cargo version + id: version + run: | + set -euo pipefail + if [[ -n "${{ inputs['cargo-version'] }}" ]]; then + echo "cargo_version=${{ inputs['cargo-version'] }}" >> "$GITHUB_OUTPUT" + else + echo "cargo_version=$(uv run python tasks/scripts/release.py get-version --cargo)" >> "$GITHUB_OUTPUT" + fi + + - name: Patch workspace version + if: steps.version.outputs.cargo_version != '' + run: | + set -euo pipefail + sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${{ steps.version.outputs.cargo_version }}"'"/}' Cargo.toml + + - name: Build ${{ steps.target.outputs.binary }} (${{ steps.target.outputs.target }}) + env: + # Preserve the release-codegen setting used by the old Dockerfile + # Rust build path so image artifacts keep the same release profile. + CARGO_PROFILE_RELEASE_CODEGEN_UNITS: "1" + run: | + set -euo pipefail + args=( + --release + --target "${{ steps.target.outputs.target }}" + -p "${{ steps.target.outputs.crate }}" + --bin "${{ steps.target.outputs.binary }}" + ) + if [[ -n "$FEATURES" ]]; then + args+=(--features "$FEATURES") + fi + if [[ -n "${{ steps.version.outputs.cargo_version }}" ]]; then + export GIT_DIR=/nonexistent + fi + mise x -- cargo build "${args[@]}" + + - name: Verify packaged binary + run: | + set -euo pipefail + BIN="target/${{ steps.target.outputs.target }}/release/${{ steps.target.outputs.binary }}" + OUTPUT="$("$BIN" --version)" + echo "$OUTPUT" + grep -q "^${{ steps.target.outputs.binary }} " <<<"$OUTPUT" + # Record glibc linkage so drift from the Ubuntu noble runtime base + # image is visible in logs. + ldd --version + ldd "$BIN" || true + + - name: Stage binary for prebuilt layout + run: | + set -euo pipefail + STAGE="prebuilt-binaries/$ARCH" + mkdir -p "$STAGE" + install -m 0755 \ + "target/${{ steps.target.outputs.target }}/release/${{ steps.target.outputs.binary }}" \ + "$STAGE/${{ steps.target.outputs.binary }}" + ls -lh "$STAGE/" + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: ${{ inputs['artifact-name'] != '' && inputs['artifact-name'] || format('rust-binary-{0}-linux-{1}', inputs.component, inputs.arch) }} + path: prebuilt-binaries/${{ inputs.arch }}/${{ steps.target.outputs.binary }} + retention-days: ${{ inputs['retention-days'] }} + if-no-files-found: error + + - name: sccache stats + if: always() + run: | + set +e + stats_bin="${SCCACHE_PATH:-sccache}" + "$stats_bin" --show-stats + status=$? + if [[ $status -ne 0 ]]; then + echo "::warning::sccache stats unavailable (exit $status)" + fi + exit 0 diff --git a/.github/workflows/shadow-shared-cpu-spike.yml b/.github/workflows/shadow-shared-cpu-spike.yml new file mode 100644 index 0000000000..ca7b2ef09b --- /dev/null +++ b/.github/workflows/shadow-shared-cpu-spike.yml @@ -0,0 +1,73 @@ +name: Shadow — Shared CPU Spike + +# OS-49 Phase 2 / PR 1 — non-blocking spike. Runs `cargo check` on the +# nv-gha-runners shared CPU pool (`linux-{amd64,arm64}-cpu8`) with a +# GHA-backed sccache. +# +# Plan, decision thresholds, and results live in the Linear doc attached +# to OS-126 ("OS-126 — Shared CPU spike plan & results"). Dispatch this +# workflow 4–5 times after merge and record numbers there. + +on: + workflow_dispatch: + +permissions: + contents: read + packages: read + +env: + CARGO_TERM_COLOR: always + CARGO_INCREMENTAL: "0" + MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Wire sccache (already RUSTC_WRAPPER in mise.toml) to the GHA cache + # backend instead of the EKS memcached used by ARC. + SCCACHE_GHA_ENABLED: "true" + +jobs: + rust-check: + name: cargo check (${{ matrix.runner }}) + strategy: + fail-fast: false + matrix: + runner: [linux-amd64-cpu8, linux-arm64-cpu8] + runs-on: ${{ matrix.runner }} + env: + # Partition the GHA sccache cache per-arch. Without this, matrix jobs + # collide on the same cache key, and the later-starting job's writes + # fail with 409 Conflict while the earlier one's writes land — leaving + # subsequent runs with asymmetric and mostly-empty caches. + # (Run 1 on 2026-04-24 showed amd64 with 0/1062 writes succeeding + # while arm64 got 575/1064.) + SCCACHE_GHA_VERSION: ${{ matrix.runner }} + container: + image: ghcr.io/nvidia/openshell/ci:latest + credentials: + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + timeout-minutes: 60 + steps: + - uses: actions/checkout@v6 + + - name: Install tools + run: mise install + + - name: Configure GHA sccache backend + # Exposes ACTIONS_CACHE_URL / ACTIONS_RUNTIME_TOKEN to subsequent steps + # so sccache (wrapped around rustc via RUSTC_WRAPPER in mise.toml) can + # initialize the GHA cache. Without this, sccache fails at startup with + # "cache url for ghac not found". The action also installs its own + # sccache binary; harmless since mise's sccache remains on PATH. + uses: mozilla-actions/sccache-action@7d986dd989559c6ecdb630a3fd2557667be217ad # v0.0.9 + + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 + with: + shared-key: shadow-shared-cpu-spike-${{ matrix.runner }} + cache-directories: .cache/sccache + + - name: cargo check + run: mise x -- cargo check --workspace --all-targets + + - name: sccache stats + if: always() + run: mise x -- sccache --show-stats diff --git a/.github/workflows/test-gpu.yml b/.github/workflows/test-gpu.yml index df953b5d31..4721c97508 100644 --- a/.github/workflows/test-gpu.yml +++ b/.github/workflows/test-gpu.yml @@ -7,57 +7,30 @@ on: workflow_dispatch: {} # Add `schedule:` here when we want nightly coverage from the same workflow. -permissions: - contents: read - pull-requests: read - packages: write +permissions: {} jobs: pr_metadata: name: Resolve PR metadata runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read outputs: should_run: ${{ steps.gate.outputs.should_run }} steps: - - id: get_pr_info - if: github.event_name == 'push' - continue-on-error: true - uses: nv-gha-runners/get-pr-info@main - + - uses: actions/checkout@v6 - id: gate - shell: bash - env: - EVENT_NAME: ${{ github.event_name }} - GITHUB_SHA_VALUE: ${{ github.sha }} - GET_PR_INFO_OUTCOME: ${{ steps.get_pr_info.outcome }} - PR_INFO: ${{ steps.get_pr_info.outputs.pr-info }} - run: | - if [ "$EVENT_NAME" != "push" ]; then - echo "should_run=true" >> "$GITHUB_OUTPUT" - exit 0 - fi - - if [ "$GET_PR_INFO_OUTCOME" != "success" ]; then - echo "should_run=false" >> "$GITHUB_OUTPUT" - exit 0 - fi - - head_sha="$(jq -r '.head.sha' <<< "$PR_INFO")" - has_gpu_label="$(jq -r '[.labels[].name] | index("test:e2e-gpu") != null' <<< "$PR_INFO")" - - # Only trust copied pull-request/* pushes that still match the PR head SHA - # and are explicitly labeled for GPU coverage. - if [ "$head_sha" = "$GITHUB_SHA_VALUE" ] && [ "$has_gpu_label" = "true" ]; then - should_run=true - else - should_run=false - fi - - echo "should_run=$should_run" >> "$GITHUB_OUTPUT" + uses: ./.github/actions/pr-gate + with: + required_label: test:e2e-gpu build-gateway: needs: [pr_metadata] if: needs.pr_metadata.outputs.should_run == 'true' + permissions: + contents: read + packages: write uses: ./.github/workflows/docker-build.yml with: component: gateway @@ -65,6 +38,9 @@ jobs: build-cluster: needs: [pr_metadata] if: needs.pr_metadata.outputs.should_run == 'true' + permissions: + contents: read + packages: write uses: ./.github/workflows/docker-build.yml with: component: cluster @@ -72,6 +48,9 @@ jobs: e2e-gpu: needs: [pr_metadata, build-gateway, build-cluster] if: needs.pr_metadata.outputs.should_run == 'true' + permissions: + contents: read + packages: read uses: ./.github/workflows/e2e-gpu-test.yaml with: image-tag: ${{ github.sha }} diff --git a/.github/workflows/test-install.yml b/.github/workflows/test-install.yml index 416adef666..06b1e007fb 100644 --- a/.github/workflows/test-install.yml +++ b/.github/workflows/test-install.yml @@ -41,7 +41,7 @@ jobs: install: fish steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - name: Install ${{ matrix.shell }} if: matrix.install diff --git a/.gitignore b/.gitignore index d6b4fa3564..1b37bfd498 100644 --- a/.gitignore +++ b/.gitignore @@ -190,6 +190,9 @@ deploy/docker/.build/ # SBOM generated output (JSON, CSV) — release artifacts, not committed deploy/sbom/output/ +# Debian package build output (default OPENSHELL_OUTPUT_DIR for tasks/scripts/package-deb.sh) +artifacts/ + # Local mise settings mise.local.toml @@ -197,8 +200,17 @@ mise.local.toml architecture/plans # Claude -.claude/settings.local.json.claude/worktrees/ +.claude/settings.local.json .claude/worktrees/ rfc.md .worktrees .z3-trace + +# RPM build artifacts +*.src.rpm +*.tar.gz +*.tar.xz +*.tar.bz2 + +# Markdown/mermaid lint tooling deps +scripts/lint-mermaid/node_modules/ diff --git a/.markdownlint-cli2.jsonc b/.markdownlint-cli2.jsonc new file mode 100644 index 0000000000..4c7f68e5a0 --- /dev/null +++ b/.markdownlint-cli2.jsonc @@ -0,0 +1,37 @@ +{ + "globs": [ + "**/*.md", + "**/*.mdx" + ], + "gitignore": true, + "ignores": [ + ".agents/**", + ".claude/**", + ".opencode/**", + ".github/**", + "THIRD-PARTY-NOTICES/**", + "CLAUDE.md", + // Man page sources use pandoc markdown with multiple H1 sections + // (NAME, SYNOPSIS, DESCRIPTION, etc.) which is standard for man + // pages but violates MD025. + "deploy/man/**" + ], + "config": { + "default": true, + // Allow long lines — prose paragraphs are single-line per project style. + "MD013": false, + // Allow GitHub-rendered HTML commonly used in READMEs (centered logos, + // collapsible sections, keyboard hints). Regular prose HTML still flagged. + "MD033": { "allowed_elements": ["p", "img", "br", "a", "div", "details", "summary", "kbd", "sub", "sup"] }, + // Allow duplicate headings in different sections. + "MD024": { "siblings_only": true }, + // Bare URLs are fine in changelogs and tables. + "MD034": false, + // Internal docs commonly use bare fences for diagrams and terminal sketches. + "MD040": false, + // First line does not need to be a heading. + "MD002": false, + // Repo uses padded table pipes (`| foo | bar |`); rule default is "compact". + "MD060": { "style": "padded" } + } +} diff --git a/.packit.yaml b/.packit.yaml new file mode 100644 index 0000000000..2598ae6381 --- /dev/null +++ b/.packit.yaml @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Packit configuration for OpenShell RPM builds via Fedora COPR. +# See https://packit.dev/docs/configuration for full reference. + +upstream_tag_template: "v{version}" +upstream_package_name: openshell +downstream_package_name: openshell +specfile_path: openshell.spec + +# Packages needed in the SRPM build environment to create vendor tarball +srpm_build_deps: + - rust + - cargo + - git-core + +actions: + get-current-version: + # Derive version from the latest SemVer upstream tag on the current branch. + # Avoid operational tags such as vm-dev; Packit normalizes that to m.dev, + # which is not a valid Cargo package version. + - 'bash -c "git describe --tags --match ''v[0-9]*.[0-9]*.[0-9]*'' --abbrev=0 HEAD | sed ''s/^v//''"' + + create-archive: + # Step 1: Create source tarball from git working tree. + # Uses git ls-files + tar instead of git archive so the tarball + # reflects any patching that Packit may have done (e.g. version bumps). + - 'bash -c "VERSION=${PACKIT_PROJECT_VERSION} && TMPDIR=$(mktemp -d) && DIR=openshell-${VERSION} && mkdir -p ${TMPDIR}/${DIR} && git ls-files -z | xargs -0 tar cf - | tar xf - -C ${TMPDIR}/${DIR}/ && tar -czf openshell-${VERSION}.tar.gz -C ${TMPDIR} ${DIR} && rm -rf ${TMPDIR}"' + # Step 2: Create vendored Cargo dependencies tarball for offline RPM build. + - 'bash -c "VERSION=${PACKIT_PROJECT_VERSION} && CARGO_HTTP_TIMEOUT=600 CARGO_NET_RETRY=5 cargo vendor --locked --quiet && tar -cJf openshell-${VERSION}-vendor.tar.xz vendor/ && rm -rf vendor/"' + # Step 3: Return the primary archive name. Packit expects create-archive + # to print one path for Source0; Source1 is patched explicitly below. + - 'bash -c "echo openshell-${PACKIT_PROJECT_VERSION}.tar.gz"' + + fix-spec-file: + # Update Source0 to the generated tarball name + - 'bash -c "sed -i \"s|^Source0:.*|Source0: openshell-${PACKIT_PROJECT_VERSION}.tar.gz|\" openshell.spec"' + # Update Source1 to the generated vendor tarball name + - 'bash -c "sed -i \"s|^Source1:.*|Source1: openshell-${PACKIT_PROJECT_VERSION}-vendor.tar.xz|\" openshell.spec"' + # Update Version + - 'bash -c "sed -i -r \"s/^Version:(\\s*)\\S+/Version:\\1${PACKIT_RPMSPEC_VERSION}/\" openshell.spec"' + # Update Release + - 'bash -c "sed -i -r \"s/^Release:(\\s*)\\S+/Release:\\1${PACKIT_RPMSPEC_RELEASE}%{?dist}/\" openshell.spec"' + +jobs: + # Build on every pull request targeting main for CI validation + - job: copr_build + trigger: pull_request + branch: main + identifier: main-pr + targets: + - fedora-all + - epel-10 + + # Build into maxamillion/openshell on every commit to main + # for continuous development and testing builds. + - job: copr_build + trigger: commit + branch: main + owner: "maxamillion" + project: "openshell" + identifier: main-commit + targets: + - fedora-all + - epel-10 + preserve_project: true + list_on_homepage: true + + # Build on GitHub releases for publishable RPMs. + # See: https://packit.dev/docs/configuration/upstream/copr_build#using-a-custom-copr-project + - job: copr_build + trigger: release + owner: "maxamillion" + project: "openshell" + targets: + - fedora-all + - epel-10 + preserve_project: true + list_on_homepage: true diff --git a/AGENTS.md b/AGENTS.md index a6cbd29ce8..93062fd5fb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,12 +34,15 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing | | `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints | | `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing | -| `crates/openshell-bootstrap/` | Cluster bootstrap | K3s cluster setup, image loading, mTLS PKI | +| `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, mTLS bundle storage, legacy bootstrap helpers | | `crates/openshell-ocsf/` | OCSF logging | OCSF v1.7.0 event types, builders, shorthand/JSONL formatters, tracing layers | | `crates/openshell-core/` | Shared core | Common types, configuration, error handling | | `crates/openshell-providers/` | Provider management | Credential provider backends | | `crates/openshell-tui/` | Terminal UI | Ratatui-based dashboard for monitoring | | `crates/openshell-vm/` | MicroVM runtime | Experimental, work-in-progress libkrun-based VM execution | +| `crates/openshell-driver-kubernetes/` | Kubernetes compute driver | In-process `ComputeDriver` backend for K8s sandbox pods | +| `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers | +| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) | | `python/openshell/` | Python SDK | Python bindings and CLI packaging | | `proto/` | Protobuf definitions | gRPC service contracts | | `deploy/` | Docker, Helm, K8s | Dockerfiles, Helm chart, manifests | @@ -151,7 +154,7 @@ ocsf_emit!(event); ## Sandbox Infra Changes -- If you change sandbox infrastructure, ensure `mise run sandbox` succeeds. +- If you change sandbox infrastructure, ensure the relevant sandbox e2e path succeeds. ## Commits @@ -169,7 +172,7 @@ ocsf_emit!(event); - `mise run pre-commit` — Lint, format, license headers. Run before every commit. - `mise run test` — Unit test suite. Run after code changes. -- `mise run e2e` — End-to-end tests against a running cluster. Run for infrastructure, sandbox, or policy changes. +- `mise run e2e` — End-to-end tests against a running gateway. Run for infrastructure, sandbox, or policy changes. - `mise run ci` — Full local CI (lint + compile/type checks + tests). Run before opening a PR. ## Python @@ -182,7 +185,7 @@ ocsf_emit!(event); ## Cluster Infrastructure Changes -- If you change cluster bootstrap infrastructure (e.g., `openshell-bootstrap` crate, `deploy/docker/Dockerfile.images`, `cluster-entrypoint.sh`, `cluster-healthcheck.sh`, deploy logic in `openshell-cli`), update the `debug-openshell-cluster` skill in `.agents/skills/debug-openshell-cluster/SKILL.md` to reflect those changes. +- If you change gateway deployment infrastructure (e.g., Helm values/templates, gateway image packaging, or deploy logic in `openshell-cli`), update the `debug-openshell-cluster` skill in `.agents/skills/debug-openshell-cluster/SKILL.md` to reflect those changes. ## Documentation diff --git a/CI.md b/CI.md new file mode 100644 index 0000000000..57e6627ed9 --- /dev/null +++ b/CI.md @@ -0,0 +1,113 @@ +# CI + +This document describes how OpenShell's continuous integration works for pull requests, with a focus on what contributors need to do to get their PR tested. + +For local test commands see [TESTING.md](TESTING.md). For PR conventions see [CONTRIBUTING.md](CONTRIBUTING.md). + +## Overview + +PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot mirrors trusted PR commits to internal `pull-request/` branches in this repository. The gated workflows trigger on pushes to those branches, not on the original PR. + +`Branch Checks` run automatically after copy-pr-bot mirrors the PR. E2E suites are opt-in because they are more expensive and publish temporary images. + +Two opt-in labels enable the suites: + +- `test:e2e` runs `Branch E2E Checks` (non-GPU E2E) +- `test:e2e-gpu` runs `GPU Test` + +Both are required to merge once the corresponding `E2E Gate` checks are marked required in branch protection. + +## Commit signing + +copy-pr-bot decides whether to mirror a PR automatically based on whether the author is trusted. For org members and collaborators, "trusted" means **all commits in the PR are cryptographically signed**. Unsigned commits, even from an org member, force the bot to wait for a maintainer's `/ok to test `. + +DCO sign-off (`-s` / `Signed-off-by`) is a separate requirement and does not count as commit signing. Dependabot-authored dependency update PRs are allowlisted in DCO Assistant because the bot cannot sign commits. + +### One-time setup with an SSH key + +If you already use an SSH key for `git push`, you can reuse it as a signing key. (You can also generate a separate one - GitHub allows the same SSH key as both auth and signing.) + +1. Generate a key (skip if reusing your existing SSH key): + + ```shell + ssh-keygen -t ed25519 -C "you@example.com" -f ~/.ssh/id_ed25519_signing + ``` + +2. Add the **public** key at using **New SSH key**, and set **Key type: Signing Key** (not Authentication). Signing keys are managed separately from authentication keys, even when they reuse the same key material - you have to add the entry once for each role. + +3. Configure git globally: + + ```shell + git config --global gpg.format ssh + git config --global user.signingkey ~/.ssh/id_ed25519_signing.pub + git config --global commit.gpgsign true + git config --global tag.gpgsign true + ``` + +4. Verify on a test commit: + + ```shell + git commit --allow-empty -s -m "test: signing" + ``` + + Push the branch and confirm GitHub shows the commit as **Verified**. + +## Pull request flows + +### Internal contributor PR + +Prerequisites: + +- Org member or collaborator on the repo. +- All commits cryptographically signed (see [Commit signing](#commit-signing)). +- All commits include a DCO sign-off (`git commit -s`). + +Flow: + +1. Open the PR. copy-pr-bot mirrors it to `pull-request/` automatically. +2. The mirror push runs `Branch Checks` automatically. The first `Branch E2E Checks` / `GPU Test` run only resolves metadata and skips expensive jobs unless the matching label is already set. +3. A maintainer applies `test:e2e` and/or `test:e2e-gpu`. `E2E Label Help` posts a comment with a link to the existing gated workflow run. +4. The maintainer opens that link and clicks **Re-run all jobs**. This time `pr_metadata` sees the label and the build/E2E jobs run. +5. When the run finishes, the `E2E Gate` check on the PR flips to green automatically. +6. New commits push to the mirror automatically and re-trigger `Branch Checks` plus any labeled E2E/GPU workflows. + +### Forked PR + +Prerequisites: + +- DCO sign-off (`git commit -s`) on every commit. Commit signing is not required for forks - copy-pr-bot trusts forks based on maintainer review, not signing. +- A maintainer must vouch you. See the [Vouch System](AGENTS.md#vouch-system). + +Flow: + +1. Open the PR. The vouch check confirms you are vouched (otherwise the PR is auto-closed). +2. copy-pr-bot does not mirror forks automatically. A maintainer reviews the diff and comments `/ok to test ` with your latest commit SHA. +3. After `/ok to test`, copy-pr-bot mirrors to `pull-request/`. From here the flow is identical to internal PRs: maintainer applies the label, follows the comment from `E2E Label Help`, and re-runs the workflow. + +Important: every new commit you push requires another `/ok to test ` from a maintainer before E2E will run on it. If a label is applied while the mirror is stale, `E2E Label Help` will post a comment explaining what's needed. + +## copy-pr-bot + +[copy-pr-bot](https://github.com/apps/copy-pr-bot) is a GitHub App maintained by NVIDIA that solves a specific GitHub Actions security problem: by default, `pull_request`-triggered workflows on a self-hosted runner can run an arbitrary contributor's code on hardware the project owns. For projects that need self-hosted runners (GPU access, ARM hardware, on-prem secrets), GitHub's recommended pattern is to never trigger workflows directly from external `pull_request` events. + +copy-pr-bot enforces that pattern. When a PR is opened against this repository, the bot evaluates whether the change is trusted - by default, only commits authored by org members and signed with a verified key are trusted, and forks always need an explicit per-SHA approval. Once a change passes that check, the bot mirrors the PR head into a branch named `pull-request/` inside this repository. Our self-hosted workflows then trigger on `push` to those mirror branches, never on the original `pull_request` event. + +The user-visible consequences inside this repo: + +- A PR cannot run E2E until copy-pr-bot has mirrored it. For trusted authors this happens within seconds of opening the PR; for forked PRs it requires a maintainer to comment `/ok to test `. +- New commits to a fork need a fresh `/ok to test ` before the mirror updates. +- The `pull-request/` branches are not for humans to push to - they are managed by the bot. + +The bot's full administrator documentation is internal to NVIDIA. The only command contributors may see in PR comments is `/ok to test `, used by maintainers to approve a specific commit on a forked PR for testing. + +## Workflow files + +| File | Role | +|---|---| +| `.github/workflows/branch-checks.yml` | Required non-E2E PR checks. Triggers on `push: pull-request/[0-9]+`. | +| `.github/workflows/branch-e2e.yml` | Non-GPU E2E. Triggers on `push: pull-request/[0-9]+`. | +| `.github/workflows/test-gpu.yml` | GPU E2E. Triggers on `push: pull-request/[0-9]+`. | +| `.github/actions/pr-gate/action.yml` | Composite action that resolves PR metadata and verifies the required label is set. | +| `.github/workflows/e2e-gate.yml` | Posts the required `E2E Gate` check on the PR. Re-evaluates after the gated workflow completes. | +| `.github/workflows/e2e-gate-check.yml` | Reusable gate logic shared by E2E and GPU E2E. | +| `.github/workflows/e2e-label-help.yml` | When a `test:e2e*` label is applied, posts a PR comment telling the maintainer the next manual step (re-run an existing workflow run, or `/ok to test ` to refresh the mirror). | diff --git a/CLAUDE.md b/CLAUDE.md index eef4bd20cf..43c994c2d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1 +1 @@ -@AGENTS.md \ No newline at end of file +@AGENTS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dcb3f303fe..5c091c6c41 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -41,7 +41,7 @@ All open issues are actionable — if it's in the issue tracker, it's ready to b This project ships with [agent skills](#agent-skills-for-contributors) that can diagnose problems, explore the codebase, generate policies, and walk you through common workflows. Before filing an issue: 1. Clone the repo and point your coding agent at it. -2. Load the relevant skill - `debug-openshell-cluster` for cluster problems, `debug-inference` for inference setup problems, `openshell-cli` for usage questions, `generate-sandbox-policy` for policy help. +2. Load the relevant skill - `debug-openshell-cluster` for gateway or deployment problems, `debug-inference` for inference setup problems, `openshell-cli` for usage questions, `generate-sandbox-policy` for policy help. 3. Have your agent investigate. Let it run diagnostics, read the architecture docs, and attempt a fix. 4. If the agent cannot resolve it, open an issue **with the agent's diagnostic output attached**. The issue template requires this. @@ -49,7 +49,7 @@ This project ships with [agent skills](#agent-skills-for-contributors) that can - A real bug that your agent confirmed and could not fix. - A feature proposal with a design — not a "please build this" request. -- An infrastructure problem that the `debug-openshell-cluster` skill could not resolve. +- An infrastructure problem that the gateway deployment troubleshooting skill could not resolve. - An inference setup problem that the `debug-inference` skill could not resolve. - Security vulnerabilities must follow [SECURITY.md](SECURITY.md) — **not** GitHub issues. @@ -66,7 +66,7 @@ Skills live in `.agents/skills/`. Your agent's harness can discover and load the | Category | Skill | Purpose | | --------------- | ------------------------- | --------------------------------------------------------------------------------------------------- | | Getting Started | `openshell-cli` | CLI usage, sandbox lifecycle, provider management, BYOC workflows | -| Getting Started | `debug-openshell-cluster` | Diagnose cluster startup failures and health issues | +| Getting Started | `debug-openshell-cluster` | Diagnose gateway deployment and health issues | | Getting Started | `debug-inference` | Diagnose `inference.local`, host-backed local inference, and direct external inference setup issues | | Contributing | `create-spike` | Investigate a problem, produce a structured GitHub issue | | Contributing | `build-from-issue` | Plan and implement work from a GitHub issue (maintainer workflow) | @@ -92,6 +92,7 @@ Skills connect into pipelines. Individual skill files don't describe these relat - **Policy iteration:** `openshell-cli` → `generate-sandbox-policy` Workflow state labels use the `state:*` prefix, and security work uses `topic:security`. GitHub issue templates assign built-in issue types where applicable, and agent-created issues should use issue types or manual follow-up rather than type labels. +New issues opened by users without `write`, `maintain`, or `admin` repository permission are automatically labeled `state:triage-needed` by the issue triage workflow. ## Prerequisites @@ -107,6 +108,9 @@ After installing `mise`, activate it with `mise activate` or [add it to your she Shell setup examples: ```bash +# Bash +echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc + # Fish echo '~/.local/bin/mise activate fish | source' >> ~/.config/fish/config.fish @@ -148,8 +152,8 @@ cargo build -p openshell-prover --features bundled-z3 # One-time trust mise trust -# Launch a sandbox (deploys a cluster if one isn't running) -mise run sandbox +# Run a standalone gateway for local development +mise run gateway ``` ## Building the `openshell` CLI @@ -166,32 +170,14 @@ openshell --help openshell sandbox create -- codex ``` -### Cluster debugging helpers - -Two additional scripts in `scripts/bin/` provide gateway-aware wrappers for cluster debugging: - -| Script | What it does | -| --------- | ------------------------------------------------------------------------------------ | -| `kubectl` | Runs `kubectl` inside the active gateway's k3s container via `openshell doctor exec` | -| `k9s` | Runs `k9s` inside the active gateway's k3s container via `openshell doctor exec` | - -These work for both local and remote gateways (SSH is handled automatically). Examples: - -```bash -kubectl get pods -A -kubectl logs -n openshell statefulset/openshell -k9s -k9s -n openshell -``` - ## Main Tasks These are the primary `mise` tasks for day-to-day development: | Task | Purpose | | ------------------ | ------------------------------------------------------- | -| `mise run cluster` | Bootstrap or incremental deploy | -| `mise run sandbox` | Create a sandbox on the running cluster | +| `mise run gateway` | Run a standalone gateway for local development | +| `mise run sandbox` | Create or reconnect to the dev sandbox | | `mise run test` | Default test suite | | `mise run e2e` | Default end-to-end test lane | | `mise run ci` | Full local CI checks (lint, compile/type checks, tests) | @@ -255,7 +241,7 @@ See [docs/CONTRIBUTING.mdx](docs/CONTRIBUTING.mdx) for the current docs authorin This project uses [Conventional Commits](https://www.conventionalcommits.org/). All commit messages must follow the format: -``` +```text (): [optional body] @@ -276,7 +262,7 @@ This project uses [Conventional Commits](https://www.conventionalcommits.org/). **Examples:** -``` +```text feat(cli): add --verbose flag to openshell run fix(sandbox): handle timeout errors gracefully docs: update installation instructions @@ -285,8 +271,14 @@ chore(deps): bump tokio to 1.40 ### DCO -All contributions must include a `Signed-off-by` line in each commit message. This certifies you have the right to submit the work under the project license. See the [Developer Certificate of Origin](https://developercertificate.org/). +All human contributions must include a `Signed-off-by` line in each commit message. This certifies you have the right to submit the work under the project license. See the [Developer Certificate of Origin](https://developercertificate.org/). Dependabot-authored dependency update PRs are allowlisted because the bot cannot sign commits. ```bash git commit -s -m "feat(sandbox): add new capability" ``` + +DCO sign-off is separate from cryptographic commit signing. CI requires signing for org members so that copy-pr-bot can mirror your PR automatically; see [CI.md](CI.md#commit-signing) for setup. + +## CI + +How E2E runs in CI, the `test:e2e` / `test:e2e-gpu` labels, copy-pr-bot, and commit-signing setup are documented in [CI.md](CI.md). diff --git a/Cargo.lock b/Cargo.lock index e4057f75cc..c02158d44d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -145,6 +145,17 @@ version = "1.0.102" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" +[[package]] +name = "apollo-parser" +version = "0.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "947e21ff51879f8a40d7519dfe619268de2afba4042a8a43878276de3cb910f0" +dependencies = [ + "memchr", + "rowan", + "thiserror 2.0.18", +] + [[package]] name = "argon2" version = "0.5.3" @@ -232,9 +243,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.16.2" +version = "1.16.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a054912289d18629dc78375ba2c3726a3afe3ff71b4edba9dedfca0e3446d1fc" +checksum = "0ec6fb3fe69024a75fa7e1bfb48aa6cf59706a101658ea01bfd33b2b248a038f" dependencies = [ "aws-lc-sys", "untrusted 0.7.1", @@ -243,9 +254,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.39.1" +version = "0.40.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83a25cf98105baa966497416dbd42565ce3a8cf8dbfd59803ec9ad46f3126399" +checksum = "f50037ee5e1e41e7b8f9d161680a725bd1626cb6f8c7e901f91f942850852fe7" dependencies = [ "cc", "cmake", @@ -282,9 +293,9 @@ dependencies = [ [[package]] name = "axum" -version = "0.8.8" +version = "0.8.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b52af3cb4058c895d37317bb27508dccc8e5f2d39454016b297bf4a400597b8" +checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" dependencies = [ "axum-core 0.5.6", "base64 0.22.1", @@ -309,7 +320,7 @@ dependencies = [ "sha1 0.10.6", "sync_wrapper", "tokio", - "tokio-tungstenite 0.28.0", + "tokio-tungstenite 0.29.0", "tower 0.5.3", "tower-layer", "tower-service", @@ -363,7 +374,7 @@ checksum = "b62ddb9cb1ec0a098ad4bbf9344d0713fa193ae1a80af55febcff2627b6a00c1" dependencies = [ "getrandom 0.2.17", "instant", - "rand 0.8.5", + "rand 0.8.6", ] [[package]] @@ -444,16 +455,16 @@ dependencies = [ "proc-macro2", "quote", "regex", - "rustc-hash", + "rustc-hash 2.1.2", "shlex", "syn 2.0.117", ] [[package]] name = "bitflags" -version = "2.11.0" +version = "2.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" +checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" dependencies = [ "serde_core", ] @@ -611,9 +622,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.59" +version = "1.2.60" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7a4d3ec6524d28a329fc53654bbadc9bdd7b0431f5d65f1a56ffb28a1ee5283" +checksum = "43c5703da9466b66a946814e1adf53ea2c90f10063b86290cc9eb67ce3478a20" dependencies = [ "find-msvc-tools", "jobserver", @@ -621,6 +632,12 @@ dependencies = [ "shlex", ] +[[package]] +name = "cesu8" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d43a04d8753f35258c91f8ec639f792891f748a1edbd759cf1dcea3382ad83c" + [[package]] name = "cexpr" version = "0.6.0" @@ -690,9 +707,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.6.0" +version = "4.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b193af5b67834b676abd72466a96c1024e6a6ad978a1f484bd90b85c94041351" +checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" dependencies = [ "clap_builder", "clap_derive", @@ -712,9 +729,9 @@ dependencies = [ [[package]] name = "clap_complete" -version = "4.6.0" +version = "4.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19c9f1dde76b736e3681f28cec9d5a61299cbaae0fce80a68e43724ad56031eb" +checksum = "3ff7a1dccbdd8b078c2bdebff47e404615151534d5043da397ec50286816f9cb" dependencies = [ "clap", "clap_lex", @@ -724,9 +741,9 @@ dependencies = [ [[package]] name = "clap_derive" -version = "4.6.0" +version = "4.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1110bd8a634a1ab8cb04345d8d878267d57c3cf1b38d91b71af6686408bbca6a" +checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" dependencies = [ "heck", "proc-macro2", @@ -761,6 +778,16 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" +[[package]] +name = "combine" +version = "4.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" +dependencies = [ + "bytes", + "memchr", +] + [[package]] name = "compact_str" version = "0.7.1" @@ -808,6 +835,27 @@ version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" +[[package]] +name = "const_format" +version = "0.2.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4481a617ad9a412be3b97c5d403fef8ed023103368908b9c50af598ff467cc1e" +dependencies = [ + "const_format_proc_macros", + "konst", +] + +[[package]] +name = "const_format_proc_macros" +version = "0.2.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d57c2eccfb16dbac1f4e61e206105db5820c9d26c3c472bc17c774259ef7744" +dependencies = [ + "proc-macro2", + "quote", + "unicode-xid", +] + [[package]] name = "constant_time_eq" version = "0.4.2" @@ -838,9 +886,15 @@ checksum = "0940496e5c83c54f3b753d5317daec82e8edac71c33aaa1f666d76f518de2444" dependencies = [ "hax-lib", "pastey", - "rand 0.9.2", + "rand 0.9.4", ] +[[package]] +name = "countme" +version = "3.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7704b5fdd17b18ae31c4c1da5a2e0305a2bf17b5249300a9ee9ed7b72114c636" + [[package]] name = "cpufeatures" version = "0.2.17" @@ -892,6 +946,15 @@ dependencies = [ "crossbeam-utils", ] +[[package]] +name = "crossbeam-epoch" +version = "0.9.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" +dependencies = [ + "crossbeam-utils", +] + [[package]] name = "crossbeam-queue" version = "0.3.12" @@ -1166,6 +1229,37 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "derive_builder" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947" +dependencies = [ + "derive_builder_macro", +] + +[[package]] +name = "derive_builder_core" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8" +dependencies = [ + "darling", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "derive_builder_macro" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" +dependencies = [ + "derive_builder_core", + "syn 2.0.117", +] + [[package]] name = "dialoguer" version = "0.11.0" @@ -1455,6 +1549,12 @@ version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + [[package]] name = "form_urlencoded" version = "1.2.2" @@ -1633,6 +1733,18 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "getset" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9cf0fc11e47561d47397154977bc219f4cf809b2974facc3ccb3b89e2436f912" +dependencies = [ + "proc-macro-error2", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "ghash" version = "0.5.1" @@ -1721,7 +1833,16 @@ checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ "allocator-api2", "equivalent", - "foldhash", + "foldhash 0.1.5", +] + +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "foldhash 0.2.0", ] [[package]] @@ -1837,6 +1958,15 @@ dependencies = [ "itoa", ] +[[package]] +name = "http-auth" +version = "0.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "150fa4a9462ef926824cf4519c84ed652ca8f4fbae34cb8af045b5cbcaf98822" +dependencies = [ + "memchr", +] + [[package]] name = "http-body" version = "1.0.1" @@ -1920,9 +2050,9 @@ dependencies = [ [[package]] name = "hyper-rustls" -version = "0.27.7" +version = "0.27.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" dependencies = [ "http", "hyper", @@ -1930,11 +2060,10 @@ dependencies = [ "log", "rustls", "rustls-native-certs", - "rustls-pki-types", "tokio", "tokio-rustls", "tower-service", - "webpki-roots 1.0.6", + "webpki-roots 1.0.7", ] [[package]] @@ -2301,6 +2430,50 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jni" +version = "0.21.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a87aa2bb7d2af34197c04845522473242e1aa17c12f4935d5856491a7fb8c97" +dependencies = [ + "cesu8", + "cfg-if", + "combine", + "jni-sys 0.3.1", + "log", + "thiserror 1.0.69", + "walkdir", + "windows-sys 0.45.0", +] + +[[package]] +name = "jni-sys" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41a652e1f9b6e0275df1f15b32661cf0d4b78d4d87ddec5e0c3c20f097433258" +dependencies = [ + "jni-sys 0.4.1", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -2313,9 +2486,9 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.94" +version = "0.3.95" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e04e2ef80ce82e13552136fabeef8a5ed1f985a96805761cbb9a2c34e7664d9" +checksum = "2964e92d1d9dc3364cae4d718d93f227e3abb088e747d92e0395bfdedf1c12ca" dependencies = [ "cfg-if", "futures-util", @@ -2349,6 +2522,35 @@ dependencies = [ "thiserror 1.0.69", ] +[[package]] +name = "jsonwebtoken" +version = "9.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a87cc7a48537badeae96744432de36f4be2b4a34a05a5ef32e9dd8a1c169dde" +dependencies = [ + "base64 0.22.1", + "js-sys", + "pem", + "ring", + "serde", + "serde_json", + "simple_asn1", +] + +[[package]] +name = "jsonwebtoken" +version = "10.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0529410abe238729a60b108898784df8984c87f6054c9c4fcacc47e4803c1ce1" +dependencies = [ + "base64 0.22.1", + "getrandom 0.2.17", + "js-sys", + "serde", + "serde_json", + "signature 2.2.0", +] + [[package]] name = "k8s-openapi" version = "0.21.1" @@ -2362,6 +2564,21 @@ dependencies = [ "serde_json", ] +[[package]] +name = "konst" +version = "0.2.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "128133ed7824fcd73d6e7b17957c5eb7bacb885649bd8c69708b2331a10bcefb" +dependencies = [ + "konst_macro_rules", +] + +[[package]] +name = "konst_macro_rules" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4933f3f57a8e9d9da04db23fb153356ecaf00cbd14aee46279c33dc80925c37" + [[package]] name = "kube" version = "0.90.0" @@ -2496,15 +2713,15 @@ checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" [[package]] name = "libbz2-rs-sys" -version = "0.2.2" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" +checksum = "b3a6a8c165077efc8f3a971534c50ea6a1a18b329ef4a66e897a7e3a1494565f" [[package]] name = "libc" -version = "0.2.184" +version = "0.2.185" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48f5d2a454e16a5ea0f4ced81bd44e4cfc7bd3a507b61887c99fd3538b28e4af" +checksum = "52ff2c0fe9bc6cb6b14a0592c2ff4fa9ceb83eea9db979b0487cd054946a2b8f" [[package]] name = "libcrux-intrinsics" @@ -2528,7 +2745,7 @@ dependencies = [ "libcrux-secrets", "libcrux-sha3", "libcrux-traits", - "rand 0.9.2", + "rand 0.9.4", "tls_codec", ] @@ -2569,7 +2786,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9adfd58e79d860f6b9e40e35127bfae9e5bd3ade33201d1347459011a2add034" dependencies = [ "libcrux-secrets", - "rand 0.9.2", + "rand 0.9.4", ] [[package]] @@ -2721,6 +2938,52 @@ version = "2.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" +[[package]] +name = "metrics" +version = "0.24.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d5312e9ba3771cfa961b585728215e3d972c950a3eed9252aa093d6301277e8" +dependencies = [ + "ahash", + "portable-atomic", +] + +[[package]] +name = "metrics-exporter-prometheus" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3589659543c04c7dc5526ec858591015b87cd8746583b51b48ef4353f99dbcda" +dependencies = [ + "base64 0.22.1", + "http-body-util", + "hyper", + "hyper-util", + "indexmap 2.14.0", + "ipnet", + "metrics", + "metrics-util", + "quanta", + "thiserror 2.0.18", + "tokio", + "tracing", +] + +[[package]] +name = "metrics-util" +version = "0.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdfb1365fea27e6dd9dc1dbc19f570198bc86914533ad639dae939635f096be4" +dependencies = [ + "crossbeam-epoch", + "crossbeam-utils", + "hashbrown 0.16.1", + "metrics", + "quanta", + "rand 0.9.4", + "rand_xoshiro", + "sketches-ddsketch", +] + [[package]] name = "miette" version = "7.6.0" @@ -2865,7 +3128,7 @@ checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" dependencies = [ "num-integer", "num-traits", - "rand 0.8.5", + "rand 0.8.6", ] [[package]] @@ -2879,7 +3142,7 @@ dependencies = [ "num-integer", "num-iter", "num-traits", - "rand 0.8.5", + "rand 0.8.6", "serde", "smallvec", "zeroize", @@ -2957,6 +3220,26 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3" +[[package]] +name = "oauth2" +version = "5.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d" +dependencies = [ + "base64 0.22.1", + "chrono", + "getrandom 0.2.17", + "http", + "rand 0.8.6", + "reqwest 0.12.28", + "serde", + "serde_json", + "serde_path_to_error", + "sha2 0.10.9", + "thiserror 1.0.69", + "url", +] + [[package]] name = "object" version = "0.37.3" @@ -2966,6 +3249,60 @@ dependencies = [ "memchr", ] +[[package]] +name = "oci-client" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b7f8deaffcd3b0e3baf93dddcab3d18b91d46dc37d38a8b170089b234de5bb3" +dependencies = [ + "bytes", + "chrono", + "futures-util", + "http", + "http-auth", + "jsonwebtoken 10.3.0", + "lazy_static", + "oci-spec", + "olpc-cjson", + "regex", + "reqwest 0.13.2", + "serde", + "serde_json", + "sha2 0.10.9", + "thiserror 2.0.18", + "tokio", + "tracing", + "unicase", +] + +[[package]] +name = "oci-spec" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8445a2631507cec628a15fdd6154b54a3ab3f20ed4fe9d73a3b8b7a4e1ba03a" +dependencies = [ + "const_format", + "derive_builder", + "getset", + "regex", + "serde", + "serde_json", + "strum 0.27.2", + "strum_macros 0.27.2", + "thiserror 2.0.18", +] + +[[package]] +name = "olpc-cjson" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "696183c9b5fe81a7715d074fd632e8bd46f4ccc0231a3ed7fc580a80de5f7083" +dependencies = [ + "serde", + "serde_json", + "unicode-normalization", +] + [[package]] name = "once_cell" version = "1.21.4" @@ -3009,6 +3346,7 @@ name = "openshell-cli" version = "0.0.0" dependencies = [ "anyhow", + "base64 0.22.1", "bytes", "clap", "clap_complete", @@ -3022,6 +3360,7 @@ dependencies = [ "indicatif", "miette", "nix", + "oauth2", "openshell-bootstrap", "openshell-core", "openshell-policy", @@ -3031,7 +3370,7 @@ dependencies = [ "owo-colors", "prost-types", "rcgen", - "reqwest", + "reqwest 0.12.28", "rustls", "rustls-pemfile", "serde", @@ -3045,6 +3384,7 @@ dependencies = [ "tokio-stream", "tokio-tungstenite 0.26.2", "tonic", + "tower 0.5.3", "tracing", "tracing-subscriber", "url", @@ -3068,6 +3408,23 @@ dependencies = [ "url", ] +[[package]] +name = "openshell-driver-docker" +version = "0.0.0" +dependencies = [ + "bollard", + "bytes", + "futures", + "openshell-core", + "tar", + "tempfile", + "tokio", + "tokio-stream", + "tonic", + "tracing", + "url", +] + [[package]] name = "openshell-driver-kubernetes" version = "0.0.0" @@ -3090,6 +3447,59 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "openshell-driver-podman" +version = "0.0.0" +dependencies = [ + "clap", + "futures", + "http-body-util", + "hyper", + "hyper-util", + "miette", + "nix", + "openshell-core", + "serde", + "serde_json", + "temp-env", + "thiserror 2.0.18", + "tokio", + "tokio-stream", + "tonic", + "tracing", + "tracing-subscriber", +] + +[[package]] +name = "openshell-driver-vm" +version = "0.0.0" +dependencies = [ + "bollard", + "clap", + "flate2", + "futures", + "libc", + "libloading", + "miette", + "nix", + "oci-client", + "openshell-core", + "openshell-vfio", + "polling", + "prost-types", + "serde", + "serde_json", + "sha2 0.10.9", + "tar", + "tokio", + "tokio-stream", + "tonic", + "tracing", + "tracing-subscriber", + "url", + "zstd", +] + [[package]] name = "openshell-ocsf" version = "0.0.0" @@ -3130,6 +3540,8 @@ name = "openshell-providers" version = "0.0.0" dependencies = [ "openshell-core", + "serde", + "serde_yml", "thiserror 2.0.18", ] @@ -3139,7 +3551,7 @@ version = "0.0.0" dependencies = [ "bytes", "openshell-core", - "reqwest", + "reqwest 0.12.28", "serde", "serde_json", "serde_yml", @@ -3156,10 +3568,12 @@ name = "openshell-sandbox" version = "0.0.0" dependencies = [ "anyhow", + "apollo-parser", "base64 0.22.1", "bytes", "clap", "futures", + "glob", "hex", "hmac", "ipnet", @@ -3178,6 +3592,7 @@ dependencies = [ "rustls", "rustls-pemfile", "seccompiler", + "serde", "serde_json", "serde_yml", "sha2 0.10.9", @@ -3193,7 +3608,7 @@ dependencies = [ "tracing-appender", "tracing-subscriber", "uuid", - "webpki-roots 1.0.6", + "webpki-roots 1.0.7", ] [[package]] @@ -3201,7 +3616,7 @@ name = "openshell-server" version = "0.0.0" dependencies = [ "anyhow", - "axum 0.8.8", + "axum 0.8.9", "bytes", "clap", "futures", @@ -3215,18 +3630,25 @@ dependencies = [ "hyper-rustls", "hyper-util", "ipnet", + "jsonwebtoken 9.3.1", + "metrics", + "metrics-exporter-prometheus", "miette", "openshell-core", + "openshell-driver-docker", "openshell-driver-kubernetes", + "openshell-driver-podman", + "openshell-ocsf", "openshell-policy", + "openshell-providers", "openshell-router", "petname", "pin-project-lite", "prost", "prost-types", - "rand 0.9.2", + "rand 0.9.4", "rcgen", - "reqwest", + "reqwest 0.12.28", "russh", "rustls", "rustls-pemfile", @@ -3264,37 +3686,21 @@ dependencies = [ "ratatui", "serde", "terminal-colorsaurus", - "tokio", - "tonic", - "tracing", - "url", -] - -[[package]] -name = "openshell-vm" -version = "0.0.0" -dependencies = [ - "base64 0.22.1", - "clap", - "indicatif", - "libc", - "libloading", - "miette", - "nix", - "openshell-bootstrap", - "openshell-core", - "rustls", - "rustls-pemfile", + "tokio", + "tonic", + "tracing", + "url", +] + +[[package]] +name = "openshell-vfio" +version = "0.0.0" +dependencies = [ "serde", "serde_json", - "tar", + "tempfile", "thiserror 2.0.18", - "tokio", - "tokio-rustls", - "tonic", "tracing", - "tracing-subscriber", - "zstd", ] [[package]] @@ -3381,7 +3787,7 @@ dependencies = [ "delegate", "futures", "log", - "rand 0.8.5", + "rand 0.8.6", "sha2 0.10.9", "thiserror 1.0.69", "tokio", @@ -3549,7 +3955,7 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "rand 0.8.5", + "rand 0.8.6", ] [[package]] @@ -3638,9 +4044,9 @@ dependencies = [ [[package]] name = "pkg-config" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" [[package]] name = "plain" @@ -3648,6 +4054,20 @@ version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6" +[[package]] +name = "polling" +version = "3.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d0e4f59085d47d8241c88ead0f274e8a0cb551f3625263c05eb8dd897c34218" +dependencies = [ + "cfg-if", + "concurrent-queue", + "hermit-abi", + "pin-project-lite", + "rustix 1.1.4", + "windows-sys 0.61.2", +] + [[package]] name = "poly1305" version = "0.8.0" @@ -3818,6 +4238,21 @@ dependencies = [ "autotools", ] +[[package]] +name = "quanta" +version = "0.12.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3ab5a9d756f0d97bdc89019bd2e4ea098cf9cde50ee7564dde6b81ccc8f06c7" +dependencies = [ + "crossbeam-utils", + "libc", + "once_cell", + "raw-cpuid", + "wasi", + "web-sys", + "winapi", +] + [[package]] name = "quinn" version = "0.11.9" @@ -3829,7 +4264,7 @@ dependencies = [ "pin-project-lite", "quinn-proto", "quinn-udp", - "rustc-hash", + "rustc-hash 2.1.2", "rustls", "socket2 0.6.3", "thiserror 2.0.18", @@ -3844,12 +4279,13 @@ version = "0.11.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" dependencies = [ + "aws-lc-rs", "bytes", "getrandom 0.3.4", "lru-slab", - "rand 0.9.2", + "rand 0.9.4", "ring", - "rustc-hash", + "rustc-hash 2.1.2", "rustls", "rustls-pki-types", "slab", @@ -3896,9 +4332,9 @@ checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" [[package]] name = "rand" -version = "0.8.5" +version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404" +checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a" dependencies = [ "libc", "rand_chacha 0.3.1", @@ -3907,9 +4343,9 @@ dependencies = [ [[package]] name = "rand" -version = "0.9.2" +version = "0.9.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6db2770f06117d490610c7488547d543617b21bfa07796d7a12f6f1bd53850d1" +checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" dependencies = [ "rand_chacha 0.9.0", "rand_core 0.9.5", @@ -3959,6 +4395,15 @@ version = "0.10.0-rc-3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f66ee92bc15280519ef199a274fe0cafff4245d31bc39aaa31c011ad56cb1f05" +[[package]] +name = "rand_xoshiro" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f703f4665700daf5512dcca5f43afa6af89f09db47fb56be587f80636bda2d41" +dependencies = [ + "rand_core 0.9.5", +] + [[package]] name = "ratatui" version = "0.26.3" @@ -3973,12 +4418,21 @@ dependencies = [ "lru", "paste", "stability", - "strum", + "strum 0.26.3", "unicode-segmentation", "unicode-truncate", "unicode-width 0.1.14", ] +[[package]] +name = "raw-cpuid" +version = "11.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "498cd0dc59d73224351ee52a95fee0f1a617a2eae0e7d9d720cc622c73a54186" +dependencies = [ + "bitflags", +] + [[package]] name = "rcgen" version = "0.13.2" @@ -4051,7 +4505,7 @@ dependencies = [ "msvc_spectre_libs", "num-bigint", "num-traits", - "rand 0.9.2", + "rand 0.9.4", "serde", "serde_json", "serde_yaml", @@ -4096,7 +4550,48 @@ dependencies = [ "wasm-bindgen", "wasm-bindgen-futures", "web-sys", - "webpki-roots 1.0.6", + "webpki-roots 1.0.7", +] + +[[package]] +name = "reqwest" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab3f43e3283ab1488b624b44b0e988d0acea0b3214e694730a055cb6b2efa801" +dependencies = [ + "base64 0.22.1", + "bytes", + "futures-core", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-pki-types", + "rustls-platform-verifier", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tokio-util", + "tower 0.5.3", + "tower-http 0.6.8", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "wasm-streams", + "web-sys", ] [[package]] @@ -4123,6 +4618,18 @@ dependencies = [ "windows-sys 0.52.0", ] +[[package]] +name = "rowan" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "417a3a9f582e349834051b8a10c8d71ca88da4211e4093528e36b9845f6b5f21" +dependencies = [ + "countme", + "hashbrown 0.14.5", + "rustc-hash 1.1.0", + "text-size", +] + [[package]] name = "rsa" version = "0.9.10" @@ -4206,7 +4713,7 @@ dependencies = [ "pkcs1 0.8.0-rc.4", "pkcs5", "pkcs8 0.10.2", - "rand 0.9.2", + "rand 0.9.4", "rand_core 0.10.0-rc-3", "rsa 0.10.0-rc.12", "russh-cryptovec", @@ -4255,6 +4762,12 @@ version = "0.1.27" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b50b8869d9fc858ce7266cce0194bd74df58b9d0e3f6df3a9fc8eb470d95c09d" +[[package]] +name = "rustc-hash" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" + [[package]] name = "rustc-hash" version = "2.1.2" @@ -4298,10 +4811,11 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.37" +version = "0.23.38" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4" +checksum = "69f9466fb2c14ea04357e91413efb882e2a6d4a406e625449bc0a5d360d53a21" dependencies = [ + "aws-lc-rs", "log", "once_cell", "ring", @@ -4342,12 +4856,40 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rustls-platform-verifier" +version = "0.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d99feebc72bae7ab76ba994bb5e121b8d83d910ca40b36e0921f53becc41784" +dependencies = [ + "core-foundation", + "core-foundation-sys", + "jni", + "log", + "once_cell", + "rustls", + "rustls-native-certs", + "rustls-platform-verifier-android", + "rustls-webpki", + "security-framework", + "security-framework-sys", + "webpki-root-certs", + "windows-sys 0.61.2", +] + +[[package]] +name = "rustls-platform-verifier-android" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" + [[package]] name = "rustls-webpki" -version = "0.103.10" +version = "0.103.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df33b2b81ac578cabaf06b89b0631153a3f416b0a886e8a7a1707fb51abbd1ef" +checksum = "8279bb85272c9f10811ae6a6c547ff594d6a7f3c6c6b02ee9726d1d0dcfcdd06" dependencies = [ + "aws-lc-rs", "ring", "rustls-pki-types", "untrusted 0.9.0", @@ -4374,6 +4916,15 @@ dependencies = [ "cipher", ] +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + [[package]] name = "schannel" version = "0.1.29" @@ -4751,6 +5302,24 @@ version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" +[[package]] +name = "simple_asn1" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d585997b0ac10be3c5ee635f1bab02d512760d14b7c468801ac8a01d9ae5f1d" +dependencies = [ + "num-bigint", + "num-traits", + "thiserror 2.0.18", + "time", +] + +[[package]] +name = "sketches-ddsketch" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c6f73aeb92d671e0cc4dca167e59b2deb6387c375391bc99ee743f326994a2b" + [[package]] name = "slab" version = "0.4.12" @@ -4931,7 +5500,7 @@ dependencies = [ "memchr", "once_cell", "percent-encoding", - "rand 0.8.5", + "rand 0.8.6", "rsa 0.9.10", "serde", "sha1 0.10.6", @@ -4969,7 +5538,7 @@ dependencies = [ "md-5", "memchr", "once_cell", - "rand 0.8.5", + "rand 0.8.6", "serde", "serde_json", "sha2 0.10.9", @@ -5079,9 +5648,15 @@ version = "0.26.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" dependencies = [ - "strum_macros", + "strum_macros 0.26.4", ] +[[package]] +name = "strum" +version = "0.27.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" + [[package]] name = "strum_macros" version = "0.26.4" @@ -5095,6 +5670,18 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "strum_macros" +version = "0.27.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "subtle" version = "2.6.1" @@ -5122,6 +5709,12 @@ version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b7401a30af6cb5818bb64852270bb722533397edcfc7344954a38f420819ece2" +[[package]] +name = "symlink" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7973cce6668464ea31f176d85b13c7ab3bba2cb3b77a2ed26abd7801688010a" + [[package]] name = "syn" version = "1.0.109" @@ -5233,6 +5826,12 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "text-size" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f18aa187839b2bdb1ad2fa35ead8c4c2976b64e4363c386d45ac0f7ee85c9233" + [[package]] name = "textwrap" version = "0.16.2" @@ -5372,9 +5971,9 @@ dependencies = [ [[package]] name = "tokio" -version = "1.51.1" +version = "1.52.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f66bf9585cda4b724d3e78ab34b73fb2bbaba9011b9bfdf69dc836382ea13b8c" +checksum = "b67dee974fe86fd92cc45b7a95fdd2f99a36a6d7b0d431a231178d3d670bbcc6" dependencies = [ "bytes", "libc", @@ -5437,14 +6036,14 @@ dependencies = [ [[package]] name = "tokio-tungstenite" -version = "0.28.0" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d25a406cddcc431a75d3d9afc6a7c0f7428d4891dd973e4d54c56b46127bf857" +checksum = "8f72a05e828585856dacd553fba484c242c46e391fb0e58917c942ee9202915c" dependencies = [ "futures-util", "log", "tokio", - "tungstenite 0.28.0", + "tungstenite 0.29.0", ] [[package]] @@ -5519,7 +6118,7 @@ dependencies = [ "indexmap 1.9.3", "pin-project", "pin-project-lite", - "rand 0.8.5", + "rand 0.8.6", "slab", "tokio", "tokio-util", @@ -5580,6 +6179,7 @@ dependencies = [ "tower-layer", "tower-service", "tracing", + "uuid", ] [[package]] @@ -5608,11 +6208,12 @@ dependencies = [ [[package]] name = "tracing-appender" -version = "0.2.4" +version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "786d480bce6247ab75f005b14ae1624ad978d3029d9113f0a22fa1ac773faeaf" +checksum = "050686193eb999b4bb3bc2acfa891a13da00f79734704c4b8b4ef1a10b368a3c" dependencies = [ "crossbeam-channel", + "symlink", "thiserror 2.0.18", "time", "tracing-subscriber", @@ -5698,7 +6299,7 @@ dependencies = [ "http", "httparse", "log", - "rand 0.9.2", + "rand 0.9.4", "rustls", "rustls-pki-types", "sha1 0.10.6", @@ -5708,19 +6309,18 @@ dependencies = [ [[package]] name = "tungstenite" -version = "0.28.0" +version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8628dcc84e5a09eb3d8423d6cb682965dea9133204e8fb3efee74c2a0c259442" +checksum = "6c01152af293afb9c7c2a57e4b559c5620b421f6d133261c60dd2d0cdb38e6b8" dependencies = [ "bytes", "data-encoding", "http", "httparse", "log", - "rand 0.9.2", + "rand 0.9.4", "sha1 0.10.6", "thiserror 2.0.18", - "utf-8", ] [[package]] @@ -5741,6 +6341,12 @@ version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971" +[[package]] +name = "unicase" +version = "2.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" + [[package]] name = "unicode-bidi" version = "0.3.18" @@ -5847,6 +6453,7 @@ dependencies = [ "idna", "percent-encoding", "serde", + "serde_derive", ] [[package]] @@ -5869,9 +6476,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.23.0" +version = "1.23.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ac8b6f42ead25368cf5b098aeb3dc8a1a2c05a3eee8a9a1a68c640edbfc79d9" +checksum = "ddd74a9687298c6858e9b88ec8935ec45d22e8fd5e6394fa1bd4e99a87789c76" dependencies = [ "getrandom 0.4.2", "js-sys", @@ -5896,6 +6503,16 @@ version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + [[package]] name = "want" version = "0.3.1" @@ -5913,11 +6530,11 @@ checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] name = "wasip2" -version = "1.0.2+wasi-0.2.9" +version = "1.0.3+wasi-0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9517f9239f02c069db75e65f174b3da828fe5f5b945c4dd26bd25d89c03ebcf5" +checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" dependencies = [ - "wit-bindgen", + "wit-bindgen 0.57.1", ] [[package]] @@ -5926,7 +6543,7 @@ version = "0.4.0+wasi-0.3.0-rc-2026-01-06" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" dependencies = [ - "wit-bindgen", + "wit-bindgen 0.51.0", ] [[package]] @@ -5937,9 +6554,9 @@ checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" [[package]] name = "wasm-bindgen" -version = "0.2.117" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0551fc1bb415591e3372d0bc4780db7e587d84e2a7e79da121051c5c4b89d0b0" +checksum = "0bf938a0bacb0469e83c1e148908bd7d5a6010354cf4fb73279b7447422e3a89" dependencies = [ "cfg-if", "once_cell", @@ -5950,9 +6567,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-futures" -version = "0.4.67" +version = "0.4.68" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03623de6905b7206edd0a75f69f747f134b7f0a2323392d664448bf2d3c5d87e" +checksum = "f371d383f2fb139252e0bfac3b81b265689bf45b6874af544ffa4c975ac1ebf8" dependencies = [ "js-sys", "wasm-bindgen", @@ -5960,9 +6577,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro" -version = "0.2.117" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7fbdf9a35adf44786aecd5ff89b4563a90325f9da0923236f6104e603c7e86be" +checksum = "eeff24f84126c0ec2db7a449f0c2ec963c6a49efe0698c4242929da037ca28ed" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -5970,9 +6587,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.117" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dca9693ef2bab6d4e6707234500350d8dad079eb508dca05530c85dc3a529ff2" +checksum = "9d08065faf983b2b80a79fd87d8254c409281cf7de75fc4b773019824196c904" dependencies = [ "bumpalo", "proc-macro2", @@ -5983,9 +6600,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-shared" -version = "0.2.117" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39129a682a6d2d841b6c429d0c51e5cb0ed1a03829d8b3d1e69a011e62cb3d3b" +checksum = "5fd04d9e306f1907bd13c6361b5c6bfc7b3b3c095ed3f8a9246390f8dbdee129" dependencies = [ "unicode-ident", ] @@ -6012,6 +6629,19 @@ dependencies = [ "wasmparser", ] +[[package]] +name = "wasm-streams" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1ec4f6517c9e11ae630e200b2b65d193279042e28edd4a2cda233e46670bbb" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + [[package]] name = "wasmparser" version = "0.244.0" @@ -6026,9 +6656,9 @@ dependencies = [ [[package]] name = "web-sys" -version = "0.3.94" +version = "0.3.95" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd70027e39b12f0849461e08ffc50b9cd7688d942c1c8e3c7b22273236b4dd0a" +checksum = "4f2dfbb17949fa2088e5d39408c48368947b86f7834484e87b73de55bc14d97d" dependencies = [ "js-sys", "wasm-bindgen", @@ -6044,20 +6674,29 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "webpki-root-certs" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c" +dependencies = [ + "rustls-pki-types", +] + [[package]] name = "webpki-roots" version = "0.26.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9" dependencies = [ - "webpki-roots 1.0.6", + "webpki-roots 1.0.7", ] [[package]] name = "webpki-roots" -version = "1.0.6" +version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed" +checksum = "52f5ee44c96cf55f1b349600768e3ece3a8f26010c05265ab73f945bb1a2eb9d" dependencies = [ "rustls-pki-types", ] @@ -6088,6 +6727,15 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "winapi-x86_64-pc-windows-gnu" version = "0.4.0" @@ -6195,6 +6843,15 @@ dependencies = [ "windows-link", ] +[[package]] +name = "windows-sys" +version = "0.45.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0" +dependencies = [ + "windows-targets 0.42.2", +] + [[package]] name = "windows-sys" version = "0.48.0" @@ -6240,6 +6897,21 @@ dependencies = [ "windows-link", ] +[[package]] +name = "windows-targets" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071" +dependencies = [ + "windows_aarch64_gnullvm 0.42.2", + "windows_aarch64_msvc 0.42.2", + "windows_i686_gnu 0.42.2", + "windows_i686_msvc 0.42.2", + "windows_x86_64_gnu 0.42.2", + "windows_x86_64_gnullvm 0.42.2", + "windows_x86_64_msvc 0.42.2", +] + [[package]] name = "windows-targets" version = "0.48.5" @@ -6297,6 +6969,12 @@ dependencies = [ "windows-link", ] +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8" + [[package]] name = "windows_aarch64_gnullvm" version = "0.48.5" @@ -6315,6 +6993,12 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" +[[package]] +name = "windows_aarch64_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43" + [[package]] name = "windows_aarch64_msvc" version = "0.48.5" @@ -6333,6 +7017,12 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" +[[package]] +name = "windows_i686_gnu" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f" + [[package]] name = "windows_i686_gnu" version = "0.48.5" @@ -6363,6 +7053,12 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" +[[package]] +name = "windows_i686_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060" + [[package]] name = "windows_i686_msvc" version = "0.48.5" @@ -6381,6 +7077,12 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" +[[package]] +name = "windows_x86_64_gnu" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36" + [[package]] name = "windows_x86_64_gnu" version = "0.48.5" @@ -6399,6 +7101,12 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3" + [[package]] name = "windows_x86_64_gnullvm" version = "0.48.5" @@ -6417,6 +7125,12 @@ version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" +[[package]] +name = "windows_x86_64_msvc" +version = "0.42.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" + [[package]] name = "windows_x86_64_msvc" version = "0.48.5" @@ -6467,6 +7181,12 @@ dependencies = [ "wit-bindgen-rust-macro", ] +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + [[package]] name = "wit-bindgen-core" version = "0.51.0" @@ -6620,7 +7340,7 @@ dependencies = [ "bindgen", "cmake", "pkg-config", - "reqwest", + "reqwest 0.12.28", "serde_json", "zip", ] diff --git a/Cargo.toml b/Cargo.toml index b51aee3c25..c9bfe6c91c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,6 +4,7 @@ [workspace] resolver = "2" members = ["crates/*"] +exclude = ["crates/openshell-vm"] [workspace.package] version = "0.0.0" @@ -25,7 +26,7 @@ prost-types = "0.13" # HTTP server axum = { version = "0.8", features = ["ws"] } tower = "0.5" -tower-http = { version = "0.6", features = ["cors", "trace"] } +tower-http = { version = "0.6", features = ["cors", "trace", "request-id"] } hyper = { version = "1.6", features = ["full"] } hyper-util = { version = "0.1", features = ["tokio", "server-auto"] } http = "1.2" @@ -58,6 +59,10 @@ tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] } tracing-appender = "0.2" +# Metrics +metrics = "0.24" +metrics-exporter-prometheus = { version = "0.18", default-features = false, features = ["http-listener"] } + # Unix/Process nix = { version = "0.29", features = ["signal", "process", "user", "fs", "term"] } @@ -65,6 +70,7 @@ nix = { version = "0.29", features = ["signal", "process", "user", "fs", "term"] serde = { version = "1", features = ["derive"] } serde_json = "1" serde_yml = "0.0.12" +apollo-parser = "0.8.5" # HTTP client reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } @@ -75,6 +81,12 @@ tokio-tungstenite = { version = "0.26", features = ["rustls-tls-native-roots"] } # Clipboard (OSC 52) base64 = "0.22" +# Crypto / Auth +sha2 = "0.10" +rand = "0.9" +jsonwebtoken = "9" +getrandom = "0.3" + # Filesystem embedding include_dir = "0.7" diff --git a/README.md b/README.md index aaa851452f..b773853281 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ OpenShell is the safe, private runtime for autonomous AI agents. It provides sandboxed execution environments that protect your data, credentials, and infrastructure — governed by declarative YAML policies that prevent unauthorized file access, data exfiltration, and uncontrolled network activity. -OpenShell is built agent-first. The project ships with agent skills for everything from cluster debugging to policy generation, and we expect contributors to use them. +OpenShell is built agent-first. The project ships with agent skills for everything from gateway troubleshooting to policy generation, and we expect contributors to use them. > **Alpha software — single-player mode.** OpenShell is proof-of-life: one developer, one environment, one gateway. We are building toward multi-tenant enterprise deployments, but the starting point is getting your own environment up and running. Expect rough edges. Bring your agent. @@ -16,7 +16,8 @@ OpenShell is built agent-first. The project ships with agent skills for everythi ### Prerequisites -- **Docker** — Docker Desktop (or a Docker daemon) must be running. +- **A supported host** — macOS, Windows with WSL 2, or Linux. +- **A local runtime** — Docker, Podman, or host virtualization enabled for MicroVM-backed sandboxes. ### Install @@ -40,8 +41,6 @@ Both methods install the latest stable release by default. To install a specific openshell sandbox create -- claude # or opencode, codex, copilot ``` -A gateway is created automatically on first use. To deploy on a remote host instead, pass `--remote user@host` to the create command. - The sandbox container includes the following tools by default: | Category | Tools | @@ -99,7 +98,7 @@ OpenShell isolates each sandbox in its own container with policy-enforced egress | **Policy Engine** | Enforces filesystem, network, and process constraints from application layer down to kernel. | | **Privacy Router** | Privacy-aware LLM routing that keeps sensitive context on sandbox compute. | -Under the hood, all these components run as a [K3s](https://k3s.io/) Kubernetes cluster inside a single Docker container — no separate K8s install required. The `openshell gateway` commands take care of provisioning the container and cluster. +OpenShell runs a gateway control plane that manages sandbox lifecycle through a configured compute driver. Supported compute platforms include Docker, Podman, MicroVM, and Kubernetes. ## Protection Layers @@ -128,7 +127,7 @@ OpenShell can pass host GPUs into sandboxes for local inference, fine-tuning, or openshell sandbox create --gpu --from [gpu-enabled-sandbox] -- claude ``` -The CLI auto-bootstraps a GPU-enabled gateway on first use, auto-selecting CDI when available and otherwise falling back to Docker's NVIDIA GPU request path (`--gpus all`). GPU intent is also inferred automatically for community images with `gpu` in the name. +Docker-backed GPU sandboxes auto-select CDI when available and otherwise fall back to Docker's NVIDIA GPU request path (`--gpus all`). GPU intent is also inferred automatically for community images with `gpu` in the name. **Requirements:** NVIDIA drivers and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) must be installed on the host. The sandbox image itself must include the appropriate GPU drivers and libraries for your workload — the default `base` image does not. See the [BYOC example](https://github.com/NVIDIA/OpenShell/tree/main/examples/bring-your-own-container) for building a custom sandbox image with GPU support. @@ -171,7 +170,7 @@ openshell term OpenShell Terminal UI

-The TUI gives you a live, keyboard-driven view of your cluster. Navigate with `Tab` to switch panels, `j`/`k` to move through lists, `Enter` to select, and `:` for command mode. Cluster health and sandbox status auto-refresh every two seconds. +The TUI gives you a live, keyboard-driven view of your gateway and sandboxes. Navigate with `Tab` to switch panels, `j`/`k` to move through lists, `Enter` to select, and `:` for command mode. Gateway health and sandbox status auto-refresh every two seconds. ## Community Sandboxes and BYOC @@ -195,7 +194,7 @@ cd OpenShell # Point your agent here — it will discover the skills in .agents/skills/ automatically ``` -Your agent can load skills for CLI usage (`openshell-cli`), cluster troubleshooting (`debug-openshell-cluster`), inference troubleshooting (`debug-inference`), policy generation (`generate-sandbox-policy`), and more. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full skills table. +Your agent can load skills for CLI usage (`openshell-cli`), gateway troubleshooting (`debug-openshell-cluster`), inference troubleshooting (`debug-inference`), policy generation (`generate-sandbox-policy`), and more. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full skills table. ## Built With Agents diff --git a/SECURITY.md b/SECURITY.md index 728cdb4a1d..9000efe988 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,4 +1,4 @@ -## Security +# Security NVIDIA is dedicated to the security and trust of our software products and services, including all source code repositories managed through our organization. diff --git a/TESTING.md b/TESTING.md index eba31967eb..c356b4a621 100644 --- a/TESTING.md +++ b/TESTING.md @@ -4,13 +4,13 @@ ```bash mise run test # Rust + Python unit tests -mise run e2e # End-to-end tests (requires a running cluster) +mise run e2e # End-to-end tests (starts a Docker-backed gateway) mise run ci # Everything: lint, compile checks, and tests ``` ## Test Layout -``` +```text crates/*/src/ # Inline #[cfg(test)] modules crates/*/tests/ # Rust integration tests python/openshell/ # Python unit tests (*_test.py suffix) @@ -72,8 +72,17 @@ mise run test:python # uv run pytest python/ ## E2E Tests -E2E tests run against a live cluster. `mise run e2e` deploys changed components -before running the suite. +E2E tests run against a live gateway. By default, `mise run e2e` starts an +ephemeral standalone gateway with the Docker compute driver, runs the suite, +and cleans it up afterward. To run the suite against an existing plaintext +gateway, set `OPENSHELL_GATEWAY_ENDPOINT`: + +```bash +OPENSHELL_GATEWAY_ENDPOINT=http://127.0.0.1:18080 mise run e2e +``` + +Raw endpoint mode is HTTP-only. Use a named gateway config when a gateway +requires mTLS. ### Python E2E (`e2e/python/`) @@ -125,7 +134,7 @@ def test_multiply(sandbox): | Fixture | Scope | Purpose | |---|---|---| -| `sandbox_client` | session | gRPC client connected to the active cluster | +| `sandbox_client` | session | gRPC client connected to the active gateway | | `sandbox` | function | Factory returning a `Sandbox` context manager | | `inference_client` | session | Client for managing inference routes | | `mock_inference_route` | session | Creates a mock OpenAI-protocol route for tests | @@ -168,3 +177,4 @@ The harness (`e2e/rust/src/harness/`) provides: | Variable | Purpose | |---|---| | `OPENSHELL_GATEWAY` | Override active gateway name for E2E tests | +| `OPENSHELL_GATEWAY_ENDPOINT` | Run E2E tests against an existing plaintext HTTP gateway endpoint | diff --git a/architecture/README.md b/architecture/README.md index 570fce660c..9eb109b04d 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -6,11 +6,11 @@ This project is a platform for securely running AI agents in isolated sandbox en This platform solves that problem by creating sandboxed execution environments where agents run with exactly the permissions they need and nothing more. Every sandbox is governed by a policy that defines which files the agent can access, which network hosts it can reach, and which system operations it can perform. All outbound network traffic is forced through a controlled proxy that inspects and enforces access rules in real time. -The platform packages the entire infrastructure -- orchestration gateway, sandbox runtime, networking, and Kubernetes cluster -- into a single deployable unit. A user can go from zero to a running, secured sandbox in two commands. The system handles cluster provisioning, credential management, policy enforcement, and secure remote access without requiring the user to configure Kubernetes, networking, or security policies manually. +The platform runs a gateway control plane and uses a configured compute driver to run agents in isolated sandbox environments. Supported compute platforms include Docker, Podman, Kubernetes, and the experimental MicroVM runtime. The system handles credential management, policy enforcement, and secure remote access while leaving runtime and cluster provisioning to the operator. ## How the Subsystems Fit Together -The following diagram shows how the major subsystems interact at a high level. Users interact through the CLI, which communicates with a central gateway. The gateway manages sandbox lifecycle in Kubernetes, and each sandbox enforces its own policy locally. Inference API calls to `inference.local` are routed locally within the sandbox by an embedded inference router, without traversing the gateway at request time. +The following diagram shows how the major subsystems interact at a high level. Users interact through the CLI, which communicates with a central gateway. The gateway manages sandbox lifecycle through a compute driver, and each sandbox enforces its own policy locally. Inference API calls to `inference.local` are routed locally within the sandbox by an embedded inference router, without traversing the gateway at request time. ```mermaid flowchart TB @@ -18,11 +18,12 @@ flowchart TB CLI["Command-Line Interface"] end - subgraph CLUSTER["Kubernetes Cluster (single Docker container)"] + subgraph PLATFORM["Compute Platform"] SERVER["Gateway / Control Plane"] DB["Database (SQLite or Postgres)"] + DRIVER["Compute Driver
(Docker, Podman,
Kubernetes, VM)"] - subgraph SBX["Sandbox Pod"] + subgraph SBX["Sandbox Workload"] SUPERVISOR["Sandbox Supervisor"] PROXY["Network Proxy"] ROUTER["Inference Router"] @@ -40,7 +41,8 @@ flowchart TB CLI -- "gRPC / HTTPS" --> SERVER CLI -- "SSH over HTTP CONNECT" --> SERVER SERVER -- "CRUD + Watch" --> DB - SERVER -- "Create / Delete Pods" --> SBX + SERVER -- "Create / Delete / Watch" --> DRIVER + DRIVER -- "Manage sandbox workload" --> SBX SUPERVISOR -- "Fetch Policy + Credentials + Inference Bundle" --> SERVER SUPERVISOR -- "Spawn + Restrict" --> CHILD CHILD -- "All network traffic" --> PROXY @@ -92,42 +94,41 @@ For more detail, see [Sandbox Architecture](sandbox.md) (Proxy Routing section). ### Gateway / Control Plane -The gateway is the central orchestration service. It provides the API that the CLI talks to and manages the lifecycle of sandboxes in Kubernetes. +The gateway is the central orchestration service. It provides the API that the CLI talks to and manages sandbox lifecycle through the selected compute driver. Key responsibilities: -- **Sandbox lifecycle management**: Creating, deleting, and monitoring sandboxes. When a user creates a sandbox, the gateway provisions a Kubernetes pod with the correct container image, policy, and environment configuration. +- **Sandbox lifecycle management**: Creating, deleting, and monitoring sandboxes. When a user creates a sandbox, the gateway asks the selected compute driver to provision a workload with the correct image, policy, and environment configuration. - **gRPC and HTTP APIs**: The gateway exposes a gRPC API for structured operations (sandbox CRUD, provider management, SSH session creation) and HTTP endpoints for health checks. Both protocols share a single network port through protocol multiplexing. - **Data persistence**: Sandbox records, provider credentials, SSH sessions, and inference routes are stored in a database (SQLite by default, Postgres as an option). - **TLS termination**: The gateway supports TLS with automatic protocol negotiation, so gRPC and HTTP clients can connect securely on the same port. - **SSH tunnel gateway**: The gateway provides the entry point for SSH connections into sandboxes (see Sandbox Connect below). - **Real-time updates**: The gateway streams sandbox status changes to the CLI, so users see live progress when a sandbox is starting up. -- **Inference bundle resolution**: The gateway stores cluster-level inference configuration (provider name + model ID) and resolves it into bundles containing endpoint URLs, API keys, supported protocols, provider type, and auth metadata. Sandboxes fetch these bundles at startup and refresh them periodically. The gateway does not proxy inference traffic at request time -- it only provides configuration. +- **Inference bundle resolution**: The gateway stores gateway-level inference configuration (provider name + model ID) and resolves it into bundles containing endpoint URLs, API keys, supported protocols, provider type, and auth metadata. Sandboxes fetch these bundles at startup and refresh them periodically. The gateway does not proxy inference traffic at request time -- it only provides configuration. For more detail, see [Gateway Architecture](gateway.md). -### Cluster Bootstrap and Infrastructure +### Gateway Deployment Infrastructure -The entire platform -- Kubernetes, the gateway, networking, and pre-loaded container images -- is packaged into a single Docker container. This means the only dependency a user needs is Docker. +The gateway can run as a standalone process, a container, or a Kubernetes workload installed by the Helm chart in `deploy/helm/openshell`. Operators supply the compute platform and configure the driver that the gateway should use for sandboxes. -The bootstrap system handles: +The deployment layer handles: -- **Provisioning**: Creating the Docker container with an embedded Kubernetes (k3s) cluster, pre-loaded with all required images and Helm charts. -- **Local and remote deployment**: The same bootstrap flow works for local development (Docker on the user's machine) and remote deployment (Docker on a remote host, accessed via SSH). -- **Health monitoring**: After starting the cluster, the system polls for readiness -- waiting for Kubernetes to start, for components to deploy, and for health checks to pass. -- **Credential management**: If TLS is enabled, the bootstrap process automatically extracts client certificates and stores them locally for the CLI to use. -- **Idempotent operation**: Running the deploy command again is safe. It reuses existing infrastructure or recreates only what changed. +- **Gateway startup**: Running the gateway process or installing the Kubernetes Helm release. +- **Runtime configuration**: Supplying image references, service exposure, sandbox runtime configuration, callback endpoints, and TLS material. +- **Credential distribution**: Providing TLS and SSH relay material to the gateway and sandbox workloads. +- **Compute driver configuration**: Selecting Docker, Podman, Kubernetes, or VM-backed sandbox execution. -The target onboarding experience is two commands: +The target onboarding experience is: ```bash -pip install -openshell sandbox create --remote user@host -- claude +mise run gateway:docker +openshell gateway add ``` -The first command installs the CLI. The second command bootstraps the cluster on the remote host (if needed) and launches a sandbox running the specified agent. +The first command is one local development example. Kubernetes operators use `helm upgrade --install openshell ./deploy/helm/openshell --namespace openshell` instead. The second command registers the reachable endpoint with the CLI. -For more detail, see [Cluster Bootstrap Architecture](cluster-single-node.md). +For more detail, see [Gateway Deployment and Compute Platforms](gateway-single-node.md). ### Sandbox Connect (SSH Tunneling) @@ -142,6 +143,7 @@ The connection flow works as follows: 5. The CLI and sandbox exchange SSH traffic bidirectionally through the tunnel. This design provides several benefits: + - Sandbox pods are never directly accessible from outside the cluster. - All access is authenticated and auditable through the gateway. - Session tokens can be revoked to immediately cut off access. @@ -170,7 +172,7 @@ The inference routing system transparently intercepts AI inference API calls fro **How it works end-to-end:** -1. An operator configures cluster-level inference via `openshell cluster inference set --provider --model `. This stores a reference to the named provider and model on the gateway. +1. An operator configures gateway-level inference via `openshell inference set --provider --model `. This stores a reference to the named provider and model on the gateway. 2. When a sandbox starts, the supervisor fetches an inference bundle from the gateway via the `GetInferenceBundle` RPC. The gateway resolves the stored provider reference into a complete route: endpoint URL, API key, supported protocols, provider type, and auth metadata. The sandbox refreshes this bundle eagerly in the background every 5 seconds by default (override with `OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS`). 3. The agent sends requests to `https://inference.local` using standard OpenAI or Anthropic SDK calls. 4. The sandbox proxy intercepts the HTTPS CONNECT to `inference.local` (bypassing OPA policy evaluation), TLS-terminates the connection using the sandbox's ephemeral CA, and parses the HTTP request. @@ -183,9 +185,9 @@ The inference routing system transparently intercepts AI inference API calls fro - The sandbox never sees the real API key for the backend -- credential isolation is maintained through the gateway's bundle resolution. - Routing is explicit via `inference.local`; OPA network policy is not involved in inference routing. - Provider-specific behavior (auth header style, default headers, supported protocols) is centralized in `InferenceProviderProfile` definitions in `openshell-core`. Supported inference provider types are openai, anthropic, and nvidia. -- Cluster inference is managed via CLI (`openshell cluster inference set/get`). +- Gateway inference is managed via CLI (`openshell inference set/get`). -**Inference routes** are stored on the gateway as protobuf objects (`InferenceRoute` in `proto/inference.proto`). Cluster inference uses a managed singleton route entry keyed by `inference.local` and configured from provider + model settings. Endpoint, credentials, and protocols are resolved from the referenced provider record at bundle fetch time, so rotating a provider's API key takes effect on the next bundle refresh without reconfiguring the route. +**Inference routes** are stored on the gateway as protobuf objects (`InferenceRoute` in `proto/inference.proto`). Gateway inference uses a managed singleton route entry keyed by `inference.local` and configured from provider + model settings. Endpoint, credentials, and protocols are resolved from the referenced provider record at bundle fetch time, so rotating a provider's API key takes effect on the next bundle refresh without reconfiguring the route. **Components involved:** @@ -195,21 +197,19 @@ The inference routing system transparently intercepts AI inference API calls fro | Inference pattern detection | `crates/openshell-sandbox/src/l7/inference.rs` | Matches HTTP method + path against known inference API patterns | | Local inference router | `crates/openshell-router/src/lib.rs` | Selects a compatible route by protocol and proxies to the backend | | Provider profiles | `crates/openshell-core/src/inference.rs` | Centralized auth, headers, protocols, and endpoint defaults per provider type | -| Gateway inference service | `crates/openshell-server/src/inference.rs` | Stores cluster inference config, resolves bundles with credentials from provider records | +| Gateway inference service | `crates/openshell-server/src/inference.rs` | Stores gateway inference config, resolves bundles with credentials from provider records | | Proto definitions | `proto/inference.proto` | `ClusterInferenceConfig`, `ResolvedRoute`, bundle RPCs | - ### Container and Build System -The platform produces three container images: +The platform publishes the gateway image and relies on community-maintained sandbox images: | Image | Purpose | |---|---| -| **Sandbox** | Runs inside each sandbox pod. Contains the sandbox supervisor binary, Python runtime, and agent tooling. Uses a multi-user setup (privileged supervisor, restricted agent user). | | **Gateway** | Runs the control plane. Contains the gateway binary, database migrations, and an embedded SSH client for sandbox management. | -| **Cluster** | An airgapped Kubernetes image with k3s, pre-loaded sandbox and gateway images, Helm charts, and an API gateway. This is the single container that users deploy. | +| **Sandbox** | Runs each sandbox workload. Maintained in the OpenShell Community repository or supplied by the user. | -Builds use multi-stage Dockerfiles with caching to keep rebuild times fast. A Helm chart handles Kubernetes-level configuration (service ports, health checks, security contexts, resource limits). Build automation is managed through mise tasks. +Builds use multi-stage Dockerfiles with caching to keep rebuild times fast. The Helm chart handles Kubernetes-level configuration such as service ports, health checks, security contexts, resource limits, storage, and TLS secret mounts. Docker, Podman, and VM-backed deployments configure equivalent runtime concerns through their driver-specific gateway configuration. For more detail, see [Container Management](build-containers.md). @@ -222,7 +222,7 @@ Sandbox behavior is governed by policies written in YAML and evaluated by an emb - **Process privileges**: What user/group the agent runs as. - **L7 inspection rules**: Protocol-level constraints on HTTP API calls for specific endpoints. -Inference routing to `inference.local` is configured separately at the cluster level and does not require network policy entries. The OPA engine evaluates only explicit network policies; `inference.local` connections bypass OPA entirely and are handled by the proxy's dedicated inference interception path. +Inference routing to `inference.local` is configured separately at the gateway level and does not require network policy entries. The OPA engine evaluates only explicit network policies; `inference.local` connections bypass OPA entirely and are handled by the proxy's dedicated inference interception path. Policies are not intended to be hand-edited by end users in normal operation. They are associated with sandboxes at creation time and fetched by the sandbox supervisor at startup via gRPC. For development and testing, policies can also be loaded from local files. A gateway-global policy can override all sandbox policies via `openshell policy set --global`. @@ -234,17 +234,17 @@ For more detail on the policy language, see [Policy Language](security-policy.md The CLI is the primary way users interact with the platform. It provides commands organized into four groups: -- **Gateway management** (`openshell gateway`): Deploy, stop, destroy, and inspect clusters. Supports both local and remote (SSH) targets. +- **Gateway management** (`openshell gateway`): Register, select, and inspect gateway endpoints. - **Sandbox management** (`openshell sandbox`): Create sandboxes (with optional file upload and provider auto-discovery), connect to sandboxes via SSH, and delete sandboxes. -- **Top-level commands**: `openshell status` (cluster health), `openshell logs` (sandbox logs), `openshell forward` (port forwarding), `openshell policy` (sandbox policy management), `openshell settings` (effective sandbox settings and global/sandbox key updates). +- **Top-level commands**: `openshell status` (gateway health), `openshell logs` (sandbox logs), `openshell forward` (port forwarding), `openshell policy` (sandbox policy management), `openshell settings` (effective sandbox settings and global/sandbox key updates). - **Provider management** (`openshell provider`): Create, update, list, and delete external service credentials. -- **Inference management** (`openshell cluster inference`): Configure cluster-level inference by specifying a provider and model. The gateway resolves endpoint and credential details from the named provider record. +- **Inference management** (`openshell inference`): Configure gateway-level inference by specifying a provider and model. The gateway resolves endpoint and credential details from the named provider record. The CLI resolves which gateway to operate on through a priority chain: explicit `--gateway` flag, then the `OPENSHELL_GATEWAY` environment variable, then the active gateway set by `openshell gateway select`. Gateway names are exposed to shell completion from local metadata, and `openshell gateway select` opens an interactive chooser on a TTY while falling back to a printed list in non-interactive use. The CLI supports TLS client certificates for mutual authentication with the gateway. ## How Users Get Started -The onboarding flow is designed to require minimal setup. Docker is the only prerequisite. +The onboarding flow starts from a reachable gateway endpoint. **Step 1: Install the CLI.** @@ -258,22 +258,7 @@ pip install openshell sandbox create -- claude ``` -If no cluster exists, the CLI automatically bootstraps one. It provisions a local Kubernetes cluster inside a Docker container, waits for it to become healthy, discovers the user's AI provider credentials from local configuration files, uploads them to the gateway, and launches a sandbox running the specified agent -- all from a single command. - -For remote deployment (running the sandbox on a different machine): - -```bash -openshell sandbox create --remote user@hostname -- claude -``` - -This performs the same bootstrap flow on the remote host via SSH. - -For development and testing against the current checkout, use -`scripts/remote-deploy.sh` instead. That helper syncs the local repository to -an SSH-reachable machine, builds the CLI and Docker images on the remote host, -and then runs `openshell gateway start` there. It defaults to secure gateway -startup and only enables `--plaintext`, `--disable-gateway-auth`, or -`--recreate` when explicitly requested. +Before creating a sandbox, start or deploy the gateway on the selected compute platform and register the reachable endpoint with the CLI. **Step 3: Connect to a running sandbox.** @@ -287,10 +272,10 @@ This opens an interactive SSH session into the sandbox, with all provider creden | Document | Description | |---|---| -| [Cluster Bootstrap](cluster-single-node.md) | How the platform bootstraps a Kubernetes cluster from a single Docker container, for local and remote targets. | +| [Gateway Deployment and Compute Platforms](gateway-single-node.md) | How the gateway runs across Docker, Podman, Kubernetes with Helm, and the experimental VM driver. | | [Gateway Architecture](gateway.md) | The control plane gateway: API multiplexing, gRPC services, persistence, TLS, and sandbox orchestration. | | [Gateway Communication](gateway-deploy-connect.md) | How the CLI resolves a gateway and communicates with it over mTLS, plaintext HTTP/2, or an edge-authenticated WebSocket tunnel. | -| [Gateway Security](gateway-security.md) | mTLS enforcement, PKI bootstrap, certificate hierarchy, and the gateway trust model. | +| [Gateway Security](gateway-security.md) | mTLS enforcement, PKI provisioning, certificate hierarchy, and the gateway trust model. | | [Sandbox Architecture](sandbox.md) | The sandbox execution environment: policy enforcement, Landlock, seccomp, network namespaces, and the network proxy. | | [Container Management](build-containers.md) | Container images, Dockerfiles, Helm charts, build tasks, and CI/CD. | | [Sandbox Connect](sandbox-connect.md) | SSH tunneling into sandboxes through the gateway. | @@ -299,6 +284,7 @@ This opens an interactive SSH session into the sandbox, with all provider creden | [Docs Site Architecture](docs-site.md) | Documentation source layout, navigation structure, local validation and preview workflow, and publish pipeline. | | [Policy Language](security-policy.md) | The YAML/Rego policy system that governs sandbox behavior. | | [Inference Routing](inference-routing.md) | Transparent interception and sandbox-local routing of AI inference API calls to configured backends. | +| [Docker Driver](docker-driver.md) | Docker compute driver implementation, host networking, loopback gateway connectivity. | | [System Architecture](system-architecture.md) | Top-level system architecture diagram with all deployable components and communication flows. | | [Gateway Settings Channel](gateway-settings.md) | Runtime settings channel: two-tier key-value configuration, global policy override, settings registry, CLI/TUI commands. | | [TUI](tui.md) | Terminal user interface for sandbox interaction. | diff --git a/architecture/build-containers.md b/architecture/build-containers.md index 493e7207a8..61cfe9d892 100644 --- a/architecture/build-containers.md +++ b/architecture/build-containers.md @@ -1,25 +1,60 @@ -# Container Images +# Container Images and Deployment Packaging -OpenShell produces two container images, both published for `linux/amd64` and `linux/arm64`. +OpenShell publishes the gateway image and keeps Kubernetes Helm packaging in this repository. Sandbox images are maintained in the separate OpenShell Community repository. -## Gateway (`openshell/gateway`) +## Gateway Image -The gateway runs the control plane API server. It is deployed as a StatefulSet inside the cluster container via a bundled Helm chart. +The gateway image runs the control plane API server. Kubernetes deployments use it through the Helm chart. Standalone container deployments can use the same image with driver-specific runtime configuration. - **Docker target**: `gateway` in `deploy/docker/Dockerfile.images` - **Registry**: `ghcr.io/nvidia/openshell/gateway:latest` -- **Pulled when**: Cluster startup (the Helm chart triggers the pull) -- **Entrypoint**: `openshell-server --port 8080` (gRPC + HTTP, mTLS) +- **Pulled when**: Helm install or upgrade, or standalone container deployment +- **Entrypoint**: `openshell-gateway --port 8080` -## Cluster (`openshell/cluster`) +The image contains the gateway binary and database migrations. Runtime configuration is supplied by Helm values and Kubernetes secrets for Kubernetes, or by driver-specific configuration for standalone gateway deployments. -The cluster image is a single-container Kubernetes distribution that bundles the Helm charts, Kubernetes manifests, and the `openshell-sandbox` supervisor binary needed to bootstrap the control plane. +## Helm Chart -- **Docker target**: `cluster` in `deploy/docker/Dockerfile.images` -- **Registry**: `ghcr.io/nvidia/openshell/cluster:latest` -- **Pulled when**: `openshell gateway start` +The Helm chart at `deploy/helm/openshell` owns Kubernetes deployment concerns: -The supervisor binary (`openshell-sandbox`) is built by the shared `supervisor-builder` stage in `deploy/docker/Dockerfile.images` and placed at `/opt/openshell/bin/openshell-sandbox`. It is exposed to sandbox pods at runtime via a read-only `hostPath` volume mount — it is not baked into sandbox images. +- Gateway StatefulSet and persistent volume claim. +- Service account, RBAC, and service. +- Gateway service exposure. +- TLS secret mounts and environment variables. +- Sandbox namespace, default sandbox image, and callback endpoint configuration. +- NetworkPolicy restricting sandbox SSH ingress to the gateway. + +The chart remains the supported deployment artifact for Kubernetes. + +## Image Build Pipeline + +`deploy/docker/Dockerfile.images` no longer compiles Rust. CI calls `.github/workflows/shadow-rust-native-build.yml` through `workflow_call` to build `openshell-gateway` or `openshell-sandbox` natively on the target architecture. `.github/workflows/docker-build.yml` downloads the resulting artifact, stages it at `deploy/docker/.build/prebuilt-binaries//`, builds the per-arch image with the local Buildx driver, and merges multi-arch pushes with `docker buildx imagetools create`. Callers normally publish the GitHub SHA tag, but can pass `image-tag` to publish isolated temporary tags for validation. + +Local image builds use `tasks/scripts/stage-prebuilt-binaries.sh` through `tasks/scripts/docker-build-image.sh` before invoking Docker, so clean checkouts do not need to create the staging directory manually. + +## Supervisor Delivery + +The `openshell-sandbox` supervisor is delivered by the selected compute driver: + +| Driver | Supervisor delivery | +|---|---| +| Kubernetes | Sandbox pod image or Kubernetes driver pod template configuration. | +| Docker | Local supervisor binary or supervisor image extraction configured by the gateway. | +| Podman | Read-only OCI image volume from the `supervisor-output` image. | +| VM | Embedded in the VM runtime rootfs. | + +Each compute driver owns supervisor delivery for its runtime. + +## Standalone Gateway Binary + +OpenShell also publishes a standalone `openshell-gateway` binary as a GitHub release asset. + +- **Source crate**: `crates/openshell-server` +- **Artifact name**: `openshell-gateway-.tar.gz` +- **Targets**: `x86_64-unknown-linux-gnu`, `aarch64-unknown-linux-gnu`, `aarch64-apple-darwin` +- **Release workflows**: `.github/workflows/release-dev.yml`, `.github/workflows/release-tag.yml` + +Both the standalone artifact and the deployed container image use the `openshell-gateway` binary. ## Python Wheels @@ -29,10 +64,11 @@ OpenShell also publishes Python wheels for `linux/amd64`, `linux/arm64`, and mac - There is no local Linux multiarch wheel build task. Release workflows own the per-arch Linux wheel production. - The macOS ARM64 wheel is cross-compiled with `deploy/docker/Dockerfile.python-wheels-macos` via `build:python:wheel:macos`. - Release workflows mirror the CLI layout: a Linux matrix job for amd64/arm64, a separate macOS job, and release jobs that download the per-platform wheel artifacts directly before publishing. +- Release CPU jobs run on `linux-amd64-cpu8` and `linux-arm64-cpu8`; the macOS wheel is still cross-compiled in Docker from the amd64 Linux runner. ## Sandbox Images -Sandbox images are **not built in this repository**. They are maintained in the [openshell-community](https://github.com/nvidia/openshell-community) repository and pulled from `ghcr.io/nvidia/openshell-community/sandboxes/` at runtime. +Sandbox images are not built in this repository. They are maintained in the [openshell-community](https://github.com/nvidia/openshell-community) repository and pulled from `ghcr.io/nvidia/openshell-community/sandboxes/` at runtime. The default sandbox image is `ghcr.io/nvidia/openshell-community/sandboxes/base:latest`. To use a named community sandbox: @@ -44,38 +80,14 @@ This pulls `ghcr.io/nvidia/openshell-community/sandboxes/:latest`. ## Local Development -`mise run cluster` is the primary development command. It bootstraps a cluster if one doesn't exist, then performs incremental deploys for subsequent runs. +Use the workflow that matches the driver you are changing: -The incremental deploy (`cluster-deploy-fast.sh`) fingerprints local Git changes and only rebuilds components whose files have changed: - -| Changed files | Rebuild triggered | +| Area | Typical local command | |---|---| -| Cargo manifests, proto definitions, cross-build script | Gateway + supervisor | -| `crates/openshell-server/*`, `deploy/docker/Dockerfile.images` | Gateway | -| `crates/openshell-sandbox/*`, `crates/openshell-policy/*` | Supervisor | -| `deploy/helm/openshell/*` | Helm upgrade | - -When no local changes are detected, the command is a no-op. - -**Gateway updates** are pushed to a local registry and the StatefulSet is restarted. **Supervisor updates** are copied directly into the running cluster container via `docker cp` — new sandbox pods pick up the updated binary immediately through the hostPath mount, with no image rebuild or cluster restart required. - -Fingerprints are stored in `.cache/cluster-deploy-fast.state`. You can also target specific components explicitly: - -```bash -mise run cluster -- gateway # rebuild gateway only -mise run cluster -- supervisor # rebuild supervisor only -mise run cluster -- chart # helm upgrade only -mise run cluster -- all # rebuild everything -``` - -To validate incremental routing and BuildKit cache reuse locally, run: - -```bash -mise run cluster:test:fast-deploy-cache -``` - -The harness runs isolated scenarios in temporary git worktrees, keeps its own state and cache under `.cache/cluster-deploy-fast-test/`, and writes a Markdown summary with: +| Gateway image or chart | `mise run helm:lint` and `mise run docker:build:gateway` | +| Docker driver | `mise run gateway:docker` or `mise run e2e:docker` | +| Podman driver | `mise run e2e:podman` | +| VM driver | `mise run e2e:vm` | +| Published docs | `mise run docs` | -- auto-detection checks for gateway-only, supervisor-only, shared, Helm-only, unrelated, and explicit-target changes -- cold vs warm rebuild comparisons for gateway and supervisor code changes -- container-ID invalidation coverage to verify gateway + Helm are retriggered when the cluster container changes +Kubernetes chart changes should be validated with `helm lint deploy/helm/openshell` and, when possible, by installing the chart into a disposable Kubernetes namespace. diff --git a/architecture/ci-e2e.md b/architecture/ci-e2e.md new file mode 100644 index 0000000000..4fc007fca4 --- /dev/null +++ b/architecture/ci-e2e.md @@ -0,0 +1,198 @@ +# E2E CI Architecture + +This document describes the architecture of the E2E CI flow: every workflow involved, the trigger each one listens on, why those triggers were chosen, and how the pieces fit together. For the contributor-facing how-to (labels, signing, fork flow), see [CI.md](../CI.md). + +## Goals and constraints + +Three independent goals shape the design: + +1. **Self-hosted runner safety.** Required PR checks, E2E, and GPU tests run on NVIDIA self-hosted runners. GitHub's [security hardening guide](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#hardening-for-self-hosted-runners) states bluntly: "Self-hosted runners should almost never be used for public repositories on GitHub, because any user can open pull requests against the repository and compromise the environment." Our workaround is the same one used elsewhere in NVIDIA's GHA infrastructure: copy-pr-bot mirrors trusted PRs into `pull-request/` branches inside this repository, and the self-hosted workflows trigger on `push` to those mirror branches rather than on `pull_request`. +2. **Label as a hard merge gate.** When a PR carries `test:e2e` (or `test:e2e-gpu`), the corresponding suite *must* have actually executed and passed for the PR head SHA. The label has to be enforcing, not advisory: it blocks merge unless the suite ran with the label set. +3. **Per-job least privilege on the GitHub token.** Each workflow declares `permissions: {}` at the top, and each job declares only what it needs. This follows the hardening pattern described at . + +These three goals do not compose cleanly: the safety goal forces `push: pull-request/` triggers (which the PR author can't influence), but `push` triggers don't fire on label changes, so the label gate has to come from a separate workflow on a different trigger. That is the heart of the architecture. + +## Pieces at a glance + +| File | Trigger | Role | +|---|---|---| +| `.github/copy-pr-bot.yaml` | (config) | Tells copy-pr-bot to mirror trusted PRs into `pull-request/` branches. Pre-existed. | +| `.github/workflows/branch-checks.yml` | `push: pull-request/[0-9]+` + `workflow_dispatch` | Runs required branch checks on `linux-amd64-cpu8` and `linux-arm64-cpu8`. | +| `.github/workflows/branch-e2e.yml` | `push: pull-request/[0-9]+` + `workflow_dispatch` | Runs non-GPU E2E on `linux-arm64-cpu8`. | +| `.github/workflows/test-gpu.yml` | `push: pull-request/[0-9]+` + `workflow_dispatch` | Runs GPU E2E on self-hosted GPU runners. | +| `.github/actions/pr-gate/action.yml` | (composite) | Resolves PR metadata for a `pull-request/` push and decides whether the run should proceed. Label enforcement is optional so ordinary branch checks can validate mirror metadata without introducing another PR label. | +| `.github/workflows/e2e-gate.yml` | `pull_request` + `workflow_run` | Posts the required `E2E Gate` check on the PR. Re-evaluates after the gated workflow completes. | +| `.github/workflows/e2e-gate-check.yml` | `workflow_call` | Reusable gate logic shared by E2E and GPU E2E. | +| `.github/workflows/e2e-label-help.yml` | `pull_request_target: [labeled]` | Posts a PR comment when a `test:e2e*` label is applied, telling the maintainer the next manual step (re-run an existing run, or `/ok to test ` to refresh the mirror). Does *not* dispatch the workflow itself - see "Why we don't auto-dispatch" below. | +| `.github/workflows/e2e-test.yml`, `e2e-gpu-test.yaml`, `docker-build.yml` | `workflow_call` | Reusable worker workflows. Unchanged by this design - called from the gated workflows and from release workflows. | + +## OS-49 runner migration + +OS-49 Phase 5 added non-required shadow workflows for the non-release workflows being prepared for shared-runner cutover. Phase 6 promoted the validated shared-runner path into the real non-release workflows and removed the obsolete PR-triggered shadow workflows to avoid duplicate PR checks. + +`branch-checks.yml` uses `pr-gate` without a required label. That still verifies the mirror SHA matches the source PR head SHA, but does not require a new GitHub label for ordinary required checks. `branch-e2e.yml` keeps the existing `test:e2e` gate because it publishes temporary images and runs the expensive E2E suite. `ci-image.yml` now builds amd64 and arm64 CI images natively on shared CPU runners and merges the multi-arch manifest after both per-arch images are pushed. + +The `mise-lockfile` job regenerates `mise.lock` with the CI image's pinned mise version and requires the checked-in file to match exactly. This intentionally includes generated metadata so contributors catch toolchain-version drift instead of letting different mise versions churn the lockfile. + +OS-49 Phase 7 moves the release-facing CPU jobs in `release-canary.yml`, `release-dev.yml`, and `release-tag.yml` to the same shared CPU labels. The release workflows also call `driver-vm-linux.yml`, `driver-vm-macos.yml`, and `deb-package.yml`, so those reusable workers use the same labels to avoid retaining a hidden ARC dependency in the release path. `release-vm-kernel.yml` uses the shared CPU labels for its Linux runtime and release jobs; the macOS runtime job stays on `macos-latest-xlarge` because it builds native macOS dylibs. + +## Trigger taxonomy + +Five GitHub Actions trigger types appear in this flow. Each one was chosen for a specific reason - they are not interchangeable. + +| Trigger | Workflow context | Token scope | Why we use it here | +|---|---|---|---| +| `push: pull-request/[0-9]+` | The pushed commit (mirror branch) | Repo-default | Only fires for branches copy-pr-bot created. Decouples test execution from PR author actions: the author cannot create a `pull-request/` branch themselves. | +| `pull_request` | The PR head SHA, but actions checkout the *base* branch's workflow files | Read-only for forks | Lets us post a status check on the PR's head SHA (so branch protection sees it). Used by the `E2E Gate` evaluation jobs. | +| `pull_request_target` | Base branch | Write-capable, even for forks | Needed for `e2e-label-help.yml` to post a comment on a forked PR. The workflow never checks out PR code, so the standard `pull_request_target` foot-gun does not apply. | +| `workflow_run` | Default branch | Repo-default | Fires when the gated workflow finishes. Lets us run a gate re-evaluation step in a trusted (default-branch) context. | +| `workflow_dispatch` | Caller's ref | Repo-default | Maintainer-only manual re-run (clicking "Re-run all jobs" in the Actions UI). We deliberately do not call this from another workflow - see "Why we don't auto-dispatch" below. | + +The non-obvious move here is that the same logical "did E2E pass for this PR" check has to be posted from two of these trigger contexts: a `pull_request`-triggered run (which can attach a check to the PR head SHA) and a `workflow_run`-triggered run (which knows the gated workflow finished but can only attach checks to `main`). The flow stitches them together by re-running the original `pull_request`-triggered run after the gated workflow completes. + +## Happy-path flow (trusted PR, label applied after mirror) + +```mermaid +sequenceDiagram + autonumber + participant Author as PR Author (org member) + participant GH as GitHub + participant Bot as copy-pr-bot + participant BranchE2E as Branch E2E Checks
(self-hosted) + participant Gate as E2E Gate
(github-hosted) + participant Help as E2E Label Help
(github-hosted) + participant Maintainer + + Author->>GH: Open PR (signed commits) + GH->>Bot: PR opened + Bot->>GH: push pull-request/N (mirror) + GH->>BranchE2E: push event on pull-request/N + BranchE2E->>BranchE2E: pr_metadata: should_run = false
(no label yet) + BranchE2E-->>GH: workflow concludes success
(only metadata job ran) + + GH->>Gate: pull_request opened + Gate->>Gate: no label, gate passes (no-op) + + Maintainer->>GH: apply test:e2e label + GH->>Gate: pull_request labeled + Gate->>Gate: label set,
upstream only ran metadata
→ FAIL (red) + GH->>Help: pull_request_target labeled + Help->>GH: comment on PR with link
to existing Branch E2E Checks run + Maintainer->>GH: open the linked run, click "Re-run all jobs" + GH->>BranchE2E: re-run (push event replayed) + BranchE2E->>BranchE2E: pr_metadata: should_run = true
(label set, SHA matches) + BranchE2E->>BranchE2E: build + e2e jobs run + + BranchE2E-->>GH: workflow concludes success + GH->>Gate: workflow_run completed + Gate->>GH: rerun original pull_request gate run + GH->>Gate: pull_request rerun (replays event) + Gate->>Gate: label set,
upstream success + non-gate jobs ran
→ PASS (green) +``` + +The label-help workflow is intentionally a comment-only nudge: it never dispatches the workflow itself, so the maintainer's re-run goes through the same `push`-event run-id that originally fired on the mirror. This preserves in-progress visibility on the PR's Checks tab. + +## Forked PR flow + +The shape is identical but with two extra round trips: the maintainer has to vet each commit before copy-pr-bot will mirror it. + +```mermaid +sequenceDiagram + autonumber + participant Author as PR Author (fork) + participant GH as GitHub + participant Bot as copy-pr-bot + participant Maintainer + + Author->>GH: Open PR from fork + GH->>Bot: PR opened + Bot->>Bot: not trusted, wait + Maintainer->>GH: comment "/ok to test " + Bot->>GH: push pull-request/N + Note over Bot,GH: From here, identical to the trusted flow:
label → help comment → maintainer re-runs → gate flips green + Author->>GH: push new commit + Bot->>Bot: still untrusted, wait again + Maintainer->>GH: comment "/ok to test " +``` + +## Why each design choice exists + +### Why `push` on `pull-request/` instead of `pull_request` + +`pull_request` workflows execute the workflow file from the PR's own branch. On a self-hosted runner, that means an attacker can rewrite our workflow YAML and run anything. `push: pull-request/` only fires for branches that copy-pr-bot creates, so the workflow file source is always one that the bot vetted (signed commit + trusted author, or `/ok to test`). + +### Why the gate has to verify a non-gate job actually ran + +The gated workflows always start with a `pr_metadata` job. When the label is missing, `pr_metadata` reports `should_run=false` and the build/E2E jobs are skipped. From GitHub's perspective the workflow concluded `success`. If the gate only checked top-level conclusion, an unlabeled run from earlier would satisfy the gate forever - the label could be added without ever causing E2E to actually execute. The gate's "at least one non-gate job succeeded" check (`e2e-gate-check.yml:106-110`) is what forces a re-run after labeling. + +### Why `workflow_run` is needed for the gate flip + +Once the gated workflow runs and finishes, the `pull_request`-triggered gate check posted earlier still says "fail". `workflow_run` is the only event that fires when an arbitrary other workflow completes, and it's how we know to re-evaluate the gate. But `workflow_run` runs in the *default branch context*, so a check posted from there lands on `main` instead of the PR. Workaround: instead of posting a new check, look up the most recent `pull_request`-triggered gate run for the same head SHA and call `POST /actions/runs//rerun`. The re-run replays the original `pull_request` event, so the new check posts against the PR's head SHA and branch protection picks it up. + +### Why `pull_request_target` for the label-help workflow + +A `pull_request` workflow on a forked PR receives a read-only `GITHUB_TOKEN`. That's intentional: it prevents PR-supplied workflow code from escalating. But the help workflow doesn't *run* PR code - it never checks out the PR head, only the workflow file from `main`. It needs `pull-requests: write` to post a comment. `pull_request_target` provides a write-capable token while still loading the workflow definition from `main`. The standard `pull_request_target` warning ("don't check out PR code with this token") doesn't apply because we don't check out anything. + +### Why we don't auto-dispatch the gated workflow + +An earlier iteration of this design auto-dispatched the gated workflow via `gh workflow run --ref pull-request/` from a `pull_request_target: [labeled]` workflow. It worked, but produced a worse UX: `workflow_dispatch`-triggered runs do not appear in the PR's Checks tab. The check-runs are technically attached to the PR head SHA (visible via `gh api commits//check-runs`), but the PR UI filters them out because the run isn't associated with a PR-context event. The maintainer would see "Dispatched" comment, then no progress on the PR until the gate eventually flipped from red to green many minutes later. + +We considered alternatives: + +- **Push an empty marker commit to `pull-request/` to fire a fresh `push` event.** Changes the SHA, breaks the gate's head-SHA equivalence, and writes to a branch copy-pr-bot owns. Architecturally bad. +- **Re-trigger copy-pr-bot programmatically.** copy-pr-bot only listens for `pull_request.*` and `issue_comment.created` events ([source](https://github.com/NVIDIA/gha-runners-apps/blob/main/packages/copy-pr-bot/src/app.ts)). Even commenting `/ok to test ` is a no-op when the mirror is already at that SHA - the bot calls `git.updateRef` with the same SHA and GitHub fires no new push event. There is no way to make copy-pr-bot re-fire a push without an actual SHA change. +- **Have the dispatcher post mirror Check Runs against the PR head SHA via the Checks API.** Possible, but adds a polling/webhook loop to keep the mirror checks in sync with the actual run. Not worth the complexity for a flow a maintainer goes through manually anyway. + +The current design takes the pragmatic path: when a label is applied, the help workflow posts a comment with a deep link to the existing `Branch E2E Checks` run on the mirror. The maintainer clicks **Re-run all jobs**. That re-run replays the original `push` event, so its check-runs surface on the PR's Checks tab in real time. The cost is one human click per label application, in exchange for live progress visibility. + +### Why labels and not comment commands + +Labels persist as PR metadata and survive re-runs and force-pushes. Comment-based commands like `/ok to test` don't survive the same way: a comment from yesterday doesn't enable today's commit. Branch protection rules can require a check be present; they cannot require a comment. The label is the merge gate's primary signal because it is the only thing GitHub's branch protection knows how to look at. + +## Permission posture + +The gated E2E workflows declare `permissions: {}` at the top. Branch checks and CI image publishing use the minimum workflow/job grants needed for checkout, package pulls, and package pushes. + +| Workflow | Job | Grants | +|---|---|---| +| `branch-checks.yml` | workflow default | `contents: read`, `packages: read` | +| | `pr_metadata` | `contents: read`, `pull-requests: read` | +| `ci-image.yml` | workflow default | `contents: read`, `packages: write` | +| `branch-e2e.yml`, `test-gpu.yml` | `pr_metadata` | `contents: read`, `pull-requests: read` | +| | `build-*` | `contents: read`, `packages: write` | +| | `e2e*` | `contents: read`, `packages: read` | +| `e2e-gate.yml` | `e2e`, `gpu` (`workflow_call`) | inherits via the called workflow | +| | `rerun-on-completion` | `actions: write` | +| `e2e-gate-check.yml` | `check` | `contents: read`, `pull-requests: read`, `actions: read` | +| `e2e-label-help.yml` | `hint` | `pull-requests: write`, `actions: read`, `contents: read` | + +The reusable worker workflows (`e2e-test.yml`, `e2e-gpu-test.yaml`, `docker-build.yml`) declare their own internal permissions; the calling job grants are an upper bound for them. + +Only one workflow holds an "interesting" token: `rerun-on-completion` in `e2e-gate.yml` has `actions: write`. It calls one specific endpoint - `POST /actions/runs//rerun` for an `e2e-gate.yml` run on the same head SHA - and never executes PR code. The label-help workflow holds only `pull-requests: write` for posting the comment, also without checking out PR code. + +## Release flow + +`release-tag.yml` and `release-dev.yml` call `e2e-test.yml` directly on `main` / tag pushes. Tags and `main` are inherently trusted refs, so they bypass copy-pr-bot. E2E still blocks the release jobs (`tag-ghcr-release: needs: [..., e2e]`). + +The release CPU jobs run on `linux-amd64-cpu8` and `linux-arm64-cpu8`. GitHub-hosted docs publishing and the external wheel-publish bridge keep their existing runners. VM development release workflows are tracked separately because the managed platform capability decision is still open. + +Permissions on the release workflows are not yet scoped per-job. Tracked separately. + +## Edge cases + +| Case | What happens | +|---|---| +| Label applied before copy-pr-bot mirrors the PR | Help workflow detects no `pull-request/` branch and posts a comment telling the maintainer to wait or run `/ok to test `. | +| Label applied while mirror is stale (new commit pending `/ok to test`) | Help workflow detects mirror SHA != PR head SHA and posts the corresponding comment with the SHA the maintainer needs to vet. | +| Label removed | No reaction. The next PR event (push, label, etc.) re-evaluates the gate, which now sees no label and passes as a no-op. | +| Author force-pushes after label set | copy-pr-bot re-mirrors the new SHA → gated workflow fires on `push` → because the label is still on the PR, `pr_metadata` runs the build/E2E jobs without manual re-run → `workflow_run` fires the gate re-run → new green check on the new SHA. | +| Maintainer re-runs the gated workflow manually from the Actions UI | Same as above without the force-push. This is the path the help workflow points the maintainer at. | +| Gate's first evaluation fails (label set, upstream not yet started) | Email-on-failure noise. The check eventually flips to success once upstream finishes and `workflow_run` re-runs the gate. Tracked as a known rough edge; possible fix is posting `neutral` until the upstream completes. | + +## References + +- copy-pr-bot: +- Astral hardening guidance: +- GitHub Actions security pattern for self-hosted runners: +- `pull_request_target` foot-gun: +- Contributor-facing flow doc: [../CI.md](../CI.md) diff --git a/architecture/custom-vm-runtime.md b/architecture/custom-vm-runtime.md index ce4d0bf393..9f723d8d76 100644 --- a/architecture/custom-vm-runtime.md +++ b/architecture/custom-vm-runtime.md @@ -1,123 +1,210 @@ # Custom libkrunfw VM Runtime -> Status: Experimental and work in progress (WIP). VM support is under active development and may change. +> Status: Experimental and work in progress (WIP). The VM compute driver is +> under active development and may change. ## Overview -The OpenShell gateway VM uses [libkrun](https://github.com/containers/libkrun) to boot a -lightweight microVM with Apple Hypervisor.framework (macOS) or KVM (Linux). The kernel -is embedded inside `libkrunfw`, a companion library that packages a pre-built Linux kernel. - -The stock `libkrunfw` from Homebrew ships a minimal kernel without bridge, netfilter, or -conntrack support. This is insufficient for Kubernetes pod networking. - -The custom libkrunfw runtime adds bridge CNI, iptables/nftables, and conntrack support to -the VM kernel, enabling standard Kubernetes networking. +The OpenShell gateway uses [libkrun](https://github.com/containers/libkrun) via the +`openshell-driver-vm` compute driver to boot a lightweight microVM per sandbox. +Each VM runs on Apple Hypervisor.framework (macOS) or KVM (Linux), with the guest +kernel embedded inside `libkrunfw`. + +The stock `libkrunfw` from Homebrew ships a minimal kernel without bridge, +netfilter, or conntrack support. That is insufficient for the sandbox supervisor's +per-sandbox network namespace primitives (veth pair + iptables, see +`crates/openshell-sandbox/src/sandbox/linux/netns.rs`). The custom libkrunfw +runtime adds bridge, iptables/nftables, and conntrack support to the guest +kernel. + +The driver is spawned by `openshell-gateway` as a subprocess, talks to it over a +Unix domain socket (`compute-driver.sock`) with the +`openshell.compute.v1.ComputeDriver` gRPC surface, and manages per-sandbox +microVMs. The runtime (libkrun + libkrunfw + gvproxy) and the sandbox +supervisor are embedded directly in the driver binary; each sandbox guest +rootfs is derived from a container image at create time. ## Architecture ```mermaid graph TD subgraph Host["Host (macOS / Linux)"] - BIN[openshell-vm binary] - EMB["Embedded runtime (zstd-compressed)\nlibkrun · libkrunfw · gvproxy"] - CACHE["~/.local/share/openshell/vm-runtime/{version}/"] - PROV[Runtime provenance logging] - GVP[gvproxy networking proxy] - - BIN --> EMB - BIN -->|extracts to| CACHE - BIN --> PROV - BIN -->|spawns| GVP + GATEWAY["openshell-gateway
(compute::vm::spawn)"] + DRIVER["openshell-driver-vm
(compute-driver.sock)"] + EMB["Embedded runtime (zstd)
libkrun · libkrunfw · gvproxy
+ openshell-sandbox.zst"] + GVP["gvproxy (per sandbox)
virtio-net · DHCP · DNS"] + + GATEWAY <-->|gRPC over UDS| DRIVER + DRIVER --> EMB + DRIVER -->|spawns one per sandbox| GVP end - subgraph Guest["Guest VM"] - INIT["openshell-vm-init.sh (PID 1)"] - VAL[Validates kernel capabilities] - CNI[Configures bridge CNI] - EXECA["Starts exec agent\nvsock port 10777"] - PKI[Generates mTLS PKI] - K3S[Execs k3s server] - EXECPY["openshell-vm-exec-agent.py"] - CHK["check-vm-capabilities.sh"] - - INIT --> VAL --> CNI --> EXECA --> PKI --> K3S + subgraph Guest["Per-sandbox microVM"] + SBXINIT["/srv/openshell-vm-sandbox-init.sh"] + SBX["/opt/openshell/bin/openshell-sandbox
(PID 1, supervisor)"] + SBXINIT --> SBX end - BIN -- "fork + krun_start_enter" --> INIT - GVP -- "virtio-net" --> Guest + DRIVER -- "fork + krun_start_enter" --> SBXINIT + GVP -- "virtio-net eth0" --> Guest + SBX -.->|"outbound ConnectSupervisor
gRPC stream"| GATEWAY + CLIENT["openshell-cli"] -->|SSH over supervisor relay| GATEWAY ``` +The driver spawns **one microVM per sandbox**. Each VM boots directly into +`openshell-sandbox` as PID 1. All gateway ingress — SSH, exec, connect — rides +the supervisor-initiated `ConnectSupervisor` gRPC stream opened from inside the +guest back out to the gateway, so gvproxy is configured with `-ssh-port -1` and +never binds a host-side TCP listener. + ## Embedded Runtime -The openshell-vm binary is fully self-contained, embedding both the VM runtime libraries -and a minimal rootfs as zstd-compressed byte arrays. On first use, the binary extracts -these to XDG cache directories with progress bars: +`openshell-driver-vm` embeds the VM runtime libraries and the sandbox +supervisor as zstd-compressed byte arrays, extracting on demand: -``` -~/.local/share/openshell/vm-runtime/{version}/ +```text +~/.local/share/openshell/vm-runtime// # libkrun / libkrunfw / gvproxy ├── libkrun.{dylib,so} ├── libkrunfw.{5.dylib,so.5} └── gvproxy -~/.local/share/openshell/openshell-vm/{version}/instances//rootfs/ -├── usr/local/bin/k3s -├── opt/openshell/bin/openshell-sandbox -├── opt/openshell/manifests/ -└── ... -``` - -This eliminates the need for separate bundles or downloads - a single ~120MB binary -provides everything needed to run the VM. Old cache versions are automatically -cleaned up when a new version is extracted. - -### Hybrid Approach - -The embedded rootfs uses a "minimal" configuration: -- Includes: Base Ubuntu, k3s binary, supervisor binary, helm charts, manifests -- Excludes: Pre-loaded container images (~1GB savings) - -Container images are pulled on demand when sandboxes are created. First boot takes -~30-60s as k3s initializes; subsequent boots use cached state for ~3-5s startup. - -For fully air-gapped environments requiring pre-loaded images, build with: -```bash -mise run vm:rootfs # Full rootfs (~2GB, includes images) -mise run vm:build # Rebuild binary with full rootfs +/sandboxes//rootfs/ # per-sandbox rootfs ``` -## Network Profile - -The VM uses the bridge CNI profile, which requires a custom libkrunfw with bridge and -netfilter kernel support. The init script validates these capabilities at boot and fails -fast with an actionable error if they are missing. - -### Bridge Profile - -- CNI: bridge plugin with `cni0` interface -- IP masquerade: enabled (iptables-legacy via CNI bridge plugin) -- kube-proxy: enabled (nftables mode) -- Service VIPs: functional (ClusterIP, NodePort) -- hostNetwork workarounds: not required +Old runtime cache versions are cleaned up when a new version is extracted. + +### Sandbox rootfs preparation + +Each VM sandbox starts from either a registry image fetched directly over OCI or +a local Docker image reference produced by Dockerfile-based `--from` sources. +For local Dockerfile sources, the CLI builds the image on the local Docker +daemon and passes the ordinary image tag through `template.image`. The VM driver +first checks the local Docker daemon for that tag; when present, it exports the +image filesystem and **rewrites that filesystem into a supervisor-only sandbox +guest** before caching it: + +- `/srv/openshell-vm-sandbox-init.sh` is installed as the guest entrypoint +- the bundled `openshell-sandbox` binary is copied into + `/opt/openshell/bin/openshell-sandbox` +- Kubernetes state and manifests are stripped out if the image contains them +- the guest boots directly into `openshell-sandbox` -- no Kubernetes control + plane, no kube-proxy, no CNI plugins + +See `crates/openshell-driver-vm/src/rootfs.rs` for the rewrite logic and +`crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh` for the init +script that gets installed. + +### `--internal-run-vm` helper + +The driver binary has two modes: the default mode is the gRPC server; when +launched with `--internal-run-vm` it becomes a per-sandbox launcher. The driver +spawns one launcher per sandbox as a subprocess, which in turn starts `gvproxy` +and calls `krun_start_enter` to boot the guest. Keeping the launcher in the +same binary means the driver ships a single artifact for both roles. + +When a sandbox sets `template.image` through `openshell sandbox create --from ...`, +the VM driver treats that image as the base guest rootfs source for that +sandbox. When `template.image` is omitted, the gateway fills it from the VM +driver's advertised `default_image`, which matches the gateway's configured +sandbox image. The driver: + +- resolves the image on the gateway host without Docker for registry and + community image refs +- for local Dockerfile sources, the CLI builds through the host Docker socket + and passes the resulting ordinary Docker tag through `template.image` +- unpacks the image filesystem, injects the VM sandbox init/supervisor files, + and validates required guest tools such as `bash`, `mount`, `ip`, and `sed` +- caches the prepared guest rootfs under + `/images//rootfs.tar` +- extracts a private runtime copy under + `/sandboxes//rootfs` + +The cache key uses an immutable image identity: repo digest for registry images +and the local Docker image ID for images resolved from the local daemon. +Different VM sandboxes can use different base images concurrently because the +shared cache is per image, not global for the driver. Cached prepared rootfs +entries remain on disk until the operator removes them from the VM driver state +directory. + +Docker is therefore no longer required for VM sandboxes created from registry or +community image refs. It is only required on the local CLI/gateway host when the +source is a local Dockerfile or build context. + +Local Dockerfile sources are treated as trusted local-development inputs for VM +gateways. Remote VM gateways still reject local Dockerfile sources until a +gateway-side artifact validation and transfer boundary is designed. + +There is no embedded guest rootfs fallback anymore. VM sandboxes therefore +require either `template.image` or a configured default sandbox image. This is +still replace-the-rootfs semantics, so VM images must remain base-compatible +with the sandbox guest init path. Distroless or `scratch` images are not +expected to work. + +The legacy `openshell-vm` crate remains in the repository for later +deprecation, but it is excluded from the normal workspace and release paths. +`openshell-driver-vm` owns active VM runtime build inputs. + +## Network Plane + +The driver launches a **dedicated `gvproxy` instance per sandbox** to provide the +guest's networking plane: + +- virtio-net backend over a Unix SOCK_STREAM (Linux) or SOCK_DGRAM (macOS vfkit) + socket, which surfaces as `eth0` inside the guest +- DHCP server + default router (192.168.127.1 / 192.168.127.2) for the guest's + udhcpc client +- DNS for host aliases: the guest init script seeds `/etc/hosts` with + `host.openshell.internal` → 192.168.127.1, while leaving gvproxy's legacy + `host.containers.internal` / `host.docker.internal` resolution intact + +The `-listen` API socket and the `-ssh-port` forwarder are both intentionally +omitted. After the supervisor-initiated relay migration the driver does not +enqueue any host-side port forwards, and the guest's SSH listener lives on a +Unix socket at `/run/openshell/ssh.sock` inside the VM that is reached over the +outbound `ConnectSupervisor` gRPC stream. Binding a host listener would race +concurrent sandboxes for port 2222 and surface a misleading "sshd is reachable" +endpoint. + +The sandbox supervisor's per-sandbox netns (veth pair + iptables) branches off +of this plane. libkrun's built-in TSI socket impersonation would not satisfy +those kernel-level primitives, which is why we need the custom libkrunfw. + +## Process Lifecycle Cleanup + +`openshell-driver-vm` installs a cross-platform "die when my parent dies" +primitive (`procguard`) in every link of the spawn chain so that killing +`openshell-gateway` (SIGTERM, SIGKILL, or crash) reaps the driver, per-sandbox +launcher, gvproxy, and the libkrun worker: + +- Linux: `nix::sys::prctl::set_pdeathsig(SIGKILL)` +- macOS / BSDs: `smol-rs/polling` with `ProcessOps::Exit` on a helper thread +- gvproxy (the one non-Rust child) gets `PR_SET_PDEATHSIG` via `pre_exec` on + Linux, and is SIGTERM'd from the launcher's procguard cleanup callback on + macOS + +See `crates/openshell-driver-vm/src/procguard.rs` for the implementation and +`tasks/scripts/vm/smoke-orphan-cleanup.sh` (exposed as +`mise run vm:smoke:orphan-cleanup`) for the regression test that covers both +SIGTERM and SIGKILL paths. ## Runtime Provenance -At boot, the openshell-vm binary logs provenance metadata about the loaded runtime bundle: +At driver startup the loaded runtime bundle is logged with: - Library paths and SHA-256 hashes - Whether the runtime is custom-built or stock - For custom runtimes: libkrunfw commit, kernel version, build timestamp -This information is sourced from `provenance.json` (generated by the build script) -and makes it straightforward to correlate VM behavior with a specific runtime artifact. +This information is sourced from `provenance.json` (generated by the build +script) and makes it straightforward to correlate sandbox VM behavior with a +specific runtime artifact. ## Build Pipeline ```mermaid graph LR - subgraph Source["crates/openshell-vm/runtime/"] - KCONF["kernel/openshell.kconfig\nKernel config fragment"] - README["README.md\nOperator documentation"] + subgraph Source["crates/openshell-driver-vm/runtime/"] + KCONF["kernel/openshell.kconfig
Kernel config fragment"] end subgraph Linux["Linux CI (build-libkrun.sh)"] @@ -128,145 +215,128 @@ graph LR BUILD_M["Build libkrunfw.dylib + libkrun.dylib"] end - subgraph Output["target/libkrun-build/"] - LIB_SO["libkrunfw.so + libkrun.so\n(Linux)"] - LIB_DY["libkrunfw.dylib + libkrun.dylib\n(macOS)"] + subgraph Output["vm-runtime-<platform>.tar.zst"] + LIB_SO["libkrunfw.so + libkrun.so + gvproxy
(Linux)"] + LIB_DY["libkrunfw.dylib + libkrun.dylib + gvproxy
(macOS)"] end - KCONF --> BUILD_L - BUILD_L --> LIB_SO - KCONF --> BUILD_M - BUILD_M --> LIB_DY + KCONF --> BUILD_L --> LIB_SO + KCONF --> BUILD_M --> LIB_DY ``` +The `vm-runtime-.tar.zst` artifact is consumed by +`openshell-driver-vm`'s `build.rs`, which embeds the library set into the +binary via `include_bytes!()`. Setting `OPENSHELL_VM_RUNTIME_COMPRESSED_DIR` +at build time (wired up by `tasks/scripts/gateway-vm.sh`, registered as +`mise run gateway:vm`) points the build at the staged artifacts. + ## Kernel Config Fragment -The `openshell.kconfig` fragment enables these kernel features on top of the stock -libkrunfw kernel: +The `openshell.kconfig` fragment enables these kernel features on top of the +stock libkrunfw kernel: | Feature | Key Configs | Purpose | |---------|-------------|---------| -| Network namespaces | `CONFIG_NET_NS`, `CONFIG_NAMESPACES` | Pod isolation | -| veth | `CONFIG_VETH` | Pod network namespace pairs | -| Bridge device | `CONFIG_BRIDGE`, `CONFIG_BRIDGE_NETFILTER` | cni0 bridge for pod networking, kube-proxy bridge traffic visibility | +| Network namespaces | `CONFIG_NET_NS`, `CONFIG_NAMESPACES` | Sandbox netns isolation | +| veth | `CONFIG_VETH` | Sandbox network namespace pairs | +| Bridge device | `CONFIG_BRIDGE`, `CONFIG_BRIDGE_NETFILTER` | Bridge support + iptables visibility into bridge traffic | | Netfilter framework | `CONFIG_NETFILTER`, `CONFIG_NETFILTER_ADVANCED`, `CONFIG_NETFILTER_XTABLES` | iptables/nftables framework | -| xtables match modules | `CONFIG_NETFILTER_XT_MATCH_CONNTRACK`, `_COMMENT`, `_MULTIPORT`, `_MARK`, `_STATISTIC`, `_ADDRTYPE`, `_RECENT`, `_LIMIT` | kube-proxy and kubelet iptables rules | +| xtables match modules | `CONFIG_NETFILTER_XT_MATCH_CONNTRACK`, `_COMMENT`, `_MULTIPORT`, `_MARK`, `_STATISTIC`, `_ADDRTYPE`, `_RECENT`, `_LIMIT` | Sandbox supervisor iptables rules | | Connection tracking | `CONFIG_NF_CONNTRACK`, `CONFIG_NF_CT_NETLINK` | NAT state tracking | -| NAT | `CONFIG_NF_NAT` | Service VIP DNAT/SNAT | -| iptables | `CONFIG_IP_NF_IPTABLES`, `CONFIG_IP_NF_FILTER`, `CONFIG_IP_NF_NAT`, `CONFIG_IP_NF_MANGLE` | CNI bridge masquerade and compat | -| nftables | `CONFIG_NF_TABLES`, `CONFIG_NFT_CT`, `CONFIG_NFT_NAT`, `CONFIG_NFT_MASQ`, `CONFIG_NFT_NUMGEN`, `CONFIG_NFT_FIB_IPV4` | kube-proxy nftables mode (primary) | -| IP forwarding | `CONFIG_IP_ADVANCED_ROUTER`, `CONFIG_IP_MULTIPLE_TABLES` | Pod-to-pod routing | -| IPVS | `CONFIG_IP_VS`, `CONFIG_IP_VS_RR`, `CONFIG_IP_VS_NFCT` | kube-proxy IPVS mode (optional) | -| Traffic control | `CONFIG_NET_SCH_HTB`, `CONFIG_NET_CLS_CGROUP` | Kubernetes QoS | -| Cgroups | `CONFIG_CGROUPS`, `CONFIG_CGROUP_DEVICE`, `CONFIG_MEMCG`, `CONFIG_CGROUP_PIDS` | Container resource limits | -| TUN/TAP | `CONFIG_TUN` | CNI plugin support | +| NAT | `CONFIG_NF_NAT` | Sandbox egress DNAT/SNAT | +| iptables | `CONFIG_IP_NF_IPTABLES`, `CONFIG_IP_NF_FILTER`, `CONFIG_IP_NF_NAT`, `CONFIG_IP_NF_MANGLE` | Masquerade and compat | +| nftables | `CONFIG_NF_TABLES`, `CONFIG_NFT_CT`, `CONFIG_NFT_NAT`, `CONFIG_NFT_MASQ`, `CONFIG_NFT_NUMGEN`, `CONFIG_NFT_FIB_IPV4` | nftables path | +| IP forwarding | `CONFIG_IP_ADVANCED_ROUTER`, `CONFIG_IP_MULTIPLE_TABLES` | Sandbox-to-host routing | +| Traffic control | `CONFIG_NET_SCH_HTB`, `CONFIG_NET_CLS_CGROUP` | QoS | +| Cgroups | `CONFIG_CGROUPS`, `CONFIG_CGROUP_DEVICE`, `CONFIG_MEMCG`, `CONFIG_CGROUP_PIDS` | Sandbox resource limits | +| TUN/TAP | `CONFIG_TUN` | CNI plugin compatibility; inherited from the shared kconfig, not exercised by the driver. | | Dummy interface | `CONFIG_DUMMY` | Fallback networking | -| Landlock | `CONFIG_SECURITY_LANDLOCK` | Filesystem sandboxing support | -| Seccomp filter | `CONFIG_SECCOMP_FILTER` | Syscall filtering support | +| Landlock | `CONFIG_SECURITY_LANDLOCK` | Sandbox supervisor filesystem sandboxing | +| Seccomp filter | `CONFIG_SECCOMP_FILTER` | Sandbox supervisor syscall filtering | -See `crates/openshell-vm/runtime/kernel/openshell.kconfig` for the full fragment with -inline comments explaining why each option is needed. +See `crates/openshell-driver-vm/runtime/kernel/openshell.kconfig` for the full +fragment with inline comments explaining why each option is needed. ## Verification -One verification tool is provided: - -1. **Capability checker** (`check-vm-capabilities.sh`): Runs inside the VM to verify - kernel capabilities. Produces pass/fail results for each required feature. - -## Running Commands In A Live VM - -The standalone `openshell-vm` binary supports `openshell-vm exec -- ` for a running VM. - -- Each VM instance stores local runtime state next to its instance rootfs -- libkrun maps a per-instance host Unix socket into the guest on vsock port `10777` -- `openshell-vm-init.sh` starts `openshell-vm-exec-agent.py` during boot -- `openshell-vm exec` connects to the host socket, which libkrun forwards into the guest exec agent -- The guest exec agent spawns the command, then streams stdout, stderr, and exit status back -- The host-side bootstrap also uses the exec agent to read PKI cert files from the guest - (via `cat /opt/openshell/pki/`) instead of requiring a separate vsock server - -`openshell-vm exec` also injects `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` by default so kubectl-style -commands work the same way they would inside the VM shell. +- **Capability checker** (`check-vm-capabilities.sh`): runs inside a sandbox VM + to verify kernel capabilities. Produces pass/fail results for each required + feature. +- **Orphan-cleanup smoke test**: `mise run vm:smoke:orphan-cleanup` asserts + that killing the gateway leaves zero driver, launcher, gvproxy, or libkrun + survivors. ## Build Commands -```bash +```shell # One-time setup: download pre-built runtime (~30s) mise run vm:setup -# Build and run -mise run vm - -# Build embedded binary with base rootfs (~120MB, recommended) -mise run vm:rootfs -- --base # Build base rootfs tarball -mise run vm:build # Build binary with embedded rootfs - -# Build with full rootfs (air-gapped, ~2GB+) -mise run vm:rootfs # Build full rootfs tarball -mise run vm:build # Rebuild binary +# Start openshell-gateway with the VM compute driver +mise run gateway:vm # With custom kernel (optional, adds ~20 min) -FROM_SOURCE=1 mise run vm:setup # Build runtime from source -mise run vm:build # Then build embedded binary +FROM_SOURCE=1 mise run vm:setup -# Wipe everything and start over -mise run vm:clean +# Remove the staged compressed runtime when you need a clean rebuild +rm -rf target/vm-runtime-compressed ``` +See `crates/openshell-driver-vm/README.md` for the full driver workflow, +including multi-gateway development, CLI registration, and sandbox creation +examples. + ## CI/CD -The openshell-vm build is split into two GitHub Actions workflows that publish to a -rolling `vm-dev` GitHub Release: +The driver release path is split between on-demand runtime builds and normal +OpenShell releases: ### Kernel Runtime (`release-vm-kernel.yml`) -Builds the custom libkrunfw (kernel firmware), libkrun (VMM), and gvproxy for all -supported platforms. Runs on-demand or when the kernel config / pinned versions change. +Builds the custom libkrunfw (kernel firmware), libkrun (VMM), and gvproxy for +all supported platforms. Run it on demand when the kernel config or pinned +versions change. | Platform | Runner | Build Method | |----------|--------|-------------| -| Linux ARM64 | `build-arm64` (self-hosted) | Native `build-libkrun.sh` | -| Linux x86_64 | `build-amd64` (self-hosted) | Native `build-libkrun.sh` | +| Linux ARM64 | `linux-arm64-cpu8` | Native `build-libkrun.sh` | +| Linux x86_64 | `linux-amd64-cpu8` | Native `build-libkrun.sh` | | macOS ARM64 | `macos-latest-xlarge` (GitHub-hosted) | `build-libkrun-macos.sh` | -Artifacts: `vm-runtime-{platform}.tar.zst` containing libkrun, libkrunfw, gvproxy, and -provenance metadata. +Artifacts: `vm-runtime-{platform}.tar.zst` containing libkrun, libkrunfw, +gvproxy, and provenance metadata. Each platform builds its own libkrunfw and +libkrun natively; the kernel inside libkrunfw is always Linux regardless of +host platform. The workflow publishes GitHub artifact attestations for each +runtime tarball instead of a separate runtime checksum file. -Each platform builds its own libkrunfw and libkrun natively. The kernel inside -libkrunfw is always Linux regardless of host platform. +### Driver Binary (`release-dev.yml` / `release-tag.yml`) -### VM Binary (`release-vm-dev.yml`) +Builds the self-contained `openshell-driver-vm` binary for every platform, +with the kernel runtime + bundled sandbox supervisor embedded. Development +driver binaries are published to the rolling `dev` release; tagged driver +binaries are published to the corresponding `v*` release. -Builds the self-extracting openshell-vm binary for all platforms. Runs on every push -to `main` that touches VM-related crates. +The reusable driver workflows pull the current `vm-runtime-.tar.zst` +from the `vm-runtime` release; their build jobs set +`OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed` and +run `cargo build --release -p openshell-driver-vm`. The macOS driver is +cross-compiled via osxcross (no macOS runner needed for the binary build — +only for the kernel build). -```mermaid -graph TD - CV[compute-versions] --> DL[download-kernel-runtime\nfrom vm-dev release] - DL --> RFS_ARM[build-rootfs arm64] - DL --> RFS_AMD[build-rootfs amd64] - RFS_ARM --> VM_ARM[build-vm linux-arm64] - RFS_AMD --> VM_AMD[build-vm linux-amd64] - RFS_ARM --> VM_MAC["build-vm-macos\n(osxcross, reuses arm64 rootfs)"] - VM_ARM --> REL[release-vm-dev\nupload to rolling release] - VM_AMD --> REL - VM_MAC --> REL -``` - -The macOS binary is cross-compiled via osxcross (no macOS runner needed for the binary -build — only for the kernel build). The macOS VM guest is always Linux ARM64, so it -reuses the arm64 rootfs. - -macOS binaries produced via osxcross are not codesigned. Users must self-sign: -```bash -codesign --entitlements crates/openshell-vm/entitlements.plist --force -s - ./openshell-vm -``` +macOS driver binaries produced via osxcross are not codesigned. Local +development builds are signed automatically by `tasks/scripts/gateway-vm.sh` +(registered as `mise run gateway:vm`). Release tarball users on macOS must +ad-hoc sign `openshell-driver-vm` before running VM sandboxes. ## Rollout Strategy -1. Custom runtime is embedded by default when building with `mise run vm:build`. -2. The init script validates kernel capabilities at boot and fails fast if missing. -3. For development, override with `OPENSHELL_VM_RUNTIME_DIR` to use a local directory. -4. In CI, kernel runtime is pre-built and cached in the `vm-dev` release. The binary - build downloads it via `download-kernel-runtime.sh`. +1. Custom runtime is embedded by default when building `openshell-driver-vm` + with `OPENSHELL_VM_RUNTIME_COMPRESSED_DIR` set (wired up by + `tasks/scripts/gateway-vm.sh`). +2. The sandbox init script validates kernel capabilities at boot and fails + fast if missing. +3. For development, override with `OPENSHELL_VM_RUNTIME_DIR` to use a local + directory instead of the extracted cache. +4. In CI, the kernel runtime is pre-built and cached in the `vm-runtime` release. + Dev and tagged release builds download that runtime, embed it into + `openshell-driver-vm`, and publish the driver next to `openshell-gateway`. diff --git a/architecture/docker-driver.md b/architecture/docker-driver.md new file mode 100644 index 0000000000..25d48f740a --- /dev/null +++ b/architecture/docker-driver.md @@ -0,0 +1,129 @@ +# Docker Driver + +The Docker compute driver manages sandbox containers through the local Docker +daemon using the `bollard` client. It targets local developer environments +where running a full Kubernetes cluster is unnecessary but Docker is already +available. + +The gateway remains a host process. Each sandbox container bind-mounts a Linux +`openshell-sandbox` supervisor binary and uses Docker host networking so the +supervisor can connect to a gateway that is listening on host loopback without +requiring an additional bridge-reachable listener on Linux. + +## Source Map + +| Path | Purpose | +|---|---| +| `crates/openshell-driver-docker/src/lib.rs` | Docker compute driver implementation | +| `crates/openshell-driver-docker/src/tests.rs` | Unit tests for container spec, env, TLS paths, GPU, resource limits, and cache helpers | +| `crates/openshell-server/src/cli.rs` | Gateway CLI flags for Docker driver configuration | +| `crates/openshell-server/src/lib.rs` | In-process Docker compute runtime wiring | + +## Runtime Model + +```mermaid +flowchart LR + CLI["OpenShell CLI
host"] -->|gRPC/HTTP
127.0.0.1:8080| GW["Gateway
host process"] + GW -->|Docker API| DA["Docker daemon"] + DA --> C["Sandbox container
network_mode=host"] + C --> SV["openshell-sandbox
supervisor"] + SV -->|ConnectSupervisor
OPENSHELL_ENDPOINT| GW + SV --> NS["Nested sandbox netns
workload + policy proxy"] +``` + +The Docker container itself uses `network_mode = "host"`. This is intentional +for now: it makes a gateway bound to `127.0.0.1` reachable from the supervisor +as `127.0.0.1`, matching the host process' endpoint without a bridge listener, +NAT rule, or userland proxy. + +The container also gets a Docker-managed `/etc/hosts` entry for +`host.openshell.internal` that resolves to `127.0.0.1`. This gives callers a +stable OpenShell-owned hostname for host services without requiring changes to +the host machine's hosts file. + +The supervisor still creates a nested network namespace for the actual workload +and routes workload traffic through its policy proxy. Agent network requests are +enforced by the supervisor in that nested namespace. + +## Container Spec + +`build_container_create_body()` constructs the Docker container: + +| Field | Value | Reason | +|---|---|---| +| `image` | Sandbox template image | User-selected runtime image | +| `user` | `"0"` | Supervisor needs root inside the container for namespace and mount setup | +| `entrypoint` | `/opt/openshell/bin/openshell-sandbox` | Bind-mounted supervisor binary | +| `cmd` | Empty vector | Prevents image CMD args from being appended to the supervisor entrypoint | +| `network_mode` | `"host"` | Lets supervisor connect to host loopback gateway endpoints | +| `extra_hosts` | `host.openshell.internal:127.0.0.1` | Stable container-local alias for host loopback services | +| `cap_add` | `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, `SYSLOG` | Required for supervisor isolation setup and process inspection | +| `security_opt` | `apparmor=unconfined` | Docker's default AppArmor profile blocks mount operations required by network namespace setup | +| `restart_policy` | `unless-stopped` | Resume managed sandboxes after Docker or gateway restarts | +| `device_requests` | CDI all-GPU request when `spec.gpu` is true | Enables Docker CDI GPU sandboxes when daemon support is detected | + +## Gateway Callback + +The Docker driver injects `OPENSHELL_ENDPOINT` into each sandbox container from +`Config::grpc_endpoint` without rewriting it. This is the key difference from a +bridge-network design. + +Examples: + +```shell +OPENSHELL_GRPC_ENDPOINT=http://127.0.0.1:8080 +``` + +and: + +```shell +OPENSHELL_GRPC_ENDPOINT=https://127.0.0.1:8080 +``` + +are passed into the supervisor as-is. Because the container shares the host +network namespace, `127.0.0.1` resolves to the host loopback interface and the +gateway is reachable when it binds loopback. + +The endpoint can also use the stable alias: + +```shell +OPENSHELL_GRPC_ENDPOINT=http://host.openshell.internal:8080 +``` + +In host network mode this name resolves to `127.0.0.1` inside the container. + +For TLS endpoints, the gateway certificate must include the exact endpoint host +as a subject alternative name. For `https://127.0.0.1:8080`, the certificate +needs an IP SAN for `127.0.0.1`. For `https://localhost:8080`, it needs a DNS +SAN for `localhost`. For `https://host.openshell.internal:8080`, it needs a DNS +SAN for `host.openshell.internal`. Docker sandboxes also require client TLS +material: + +| Env / flag | Purpose | +|---|---| +| `OPENSHELL_DOCKER_TLS_CA` / `--docker-tls-ca` | CA certificate mounted at `/etc/openshell/tls/client/ca.crt` | +| `OPENSHELL_DOCKER_TLS_CERT` / `--docker-tls-cert` | Client certificate mounted at `/etc/openshell/tls/client/tls.crt` | +| `OPENSHELL_DOCKER_TLS_KEY` / `--docker-tls-key` | Client private key mounted at `/etc/openshell/tls/client/tls.key` | + +When `OPENSHELL_GRPC_ENDPOINT` uses `http://`, these TLS mounts are not +required and providing them is rejected. When it uses `https://`, all three are +required. + +## Environment + +`build_environment()` merges template environment, spec environment, and +driver-controlled keys. Driver-controlled keys win: + +| Variable | Value | +|---|---| +| `OPENSHELL_ENDPOINT` | Exact configured gateway endpoint | +| `OPENSHELL_SANDBOX_ID` | Sandbox id | +| `OPENSHELL_SANDBOX` | Sandbox name | +| `OPENSHELL_SSH_SOCKET_PATH` | Unix socket path used by the supervisor's embedded SSH daemon | +| `OPENSHELL_SANDBOX_COMMAND` | `sleep infinity` | +| `OPENSHELL_TLS_CA` | Mounted CA path when HTTPS is enabled | +| `OPENSHELL_TLS_CERT` | Mounted client cert path when HTTPS is enabled | +| `OPENSHELL_TLS_KEY` | Mounted client key path when HTTPS is enabled | + +The Docker driver does not inject `OPENSHELL_SSH_HANDSHAKE_SECRET`; the +supervisor-to-gateway path relies on mTLS for the Docker callback. diff --git a/architecture/gateway-deploy-connect.md b/architecture/gateway-deploy-connect.md index ec8153302c..14bb3e90fb 100644 --- a/architecture/gateway-deploy-connect.md +++ b/architecture/gateway-deploy-connect.md @@ -99,7 +99,7 @@ This stores `auth_mode = "plaintext"`, skips mTLS certificate extraction, and by All connection artifacts are stored under `$XDG_CONFIG_HOME/openshell/` (default `~/.config/openshell/`): -``` +```text openshell/ active_gateway # plain text: active gateway name gateways/ diff --git a/architecture/gateway-security.md b/architecture/gateway-security.md index 4989f69b6f..b8c00571d8 100644 --- a/architecture/gateway-security.md +++ b/architecture/gateway-security.md @@ -2,17 +2,17 @@ ## Overview -By default, communication with the OpenShell gateway is secured by mutual TLS (mTLS). The CLI, SDK, and sandbox pods present certificates signed by the cluster CA before they reach any application handler. The PKI is bootstrapped automatically during cluster deployment, and certificates are distributed to Kubernetes secrets and the local filesystem without manual configuration. +By default, communication with the OpenShell gateway is secured by mutual TLS (mTLS). The CLI, SDK, and sandbox workloads present certificates signed by the deployment CA before they reach any application handler. In Helm deployments, operators provide the certificate bundle as Kubernetes secrets and place the CLI client bundle in the local gateway credential directory. Non-Kubernetes deployments provide equivalent certificate files to the gateway and sandbox runtime. The gateway also supports Cloudflare-fronted deployments where the edge, not the gateway, is the first authentication boundary. In that mode the gateway either keeps TLS enabled but allows no-certificate client handshakes (`allow_unauthenticated=true`) and relies on application-layer Cloudflare JWTs, or disables gateway TLS entirely and serves plaintext behind a trusted reverse proxy or tunnel. -This document covers the certificate hierarchy, the bootstrap process, how gateway transport security modes are enforced, how sandboxes and the CLI consume their certificates, and the broader security model of the gateway. +This document covers the certificate hierarchy, how gateway transport security modes are enforced, how sandboxes and the CLI consume their certificates, and the broader security model of the gateway. ## Architecture Diagram ```mermaid graph TD - subgraph PKI["PKI (generated at bootstrap)"] + subgraph PKI["PKI (operator provided)"] CA["openshell-ca
(self-signed root)"] SERVER_CERT["openshell-server cert
(signed by CA)"] CLIENT_CERT["openshell-client cert
(signed by CA, shared)"] @@ -20,17 +20,17 @@ graph TD CA --> CLIENT_CERT end - subgraph CLUSTER["Kubernetes Cluster"] + subgraph CLUSTER["Kubernetes Helm Deployment"] S1["openshell-server-tls
Secret (server cert+key)"] S2["openshell-server-client-ca
Secret (CA cert)"] S3["openshell-client-tls
Secret (client cert+key+CA)"] GW["Gateway Process
(tokio-rustls)"] - SBX["Sandbox Pod"] + SBX["Sandbox Workload"] end subgraph HOST["User's Machine"] CLI["CLI"] - MTLS_DIR["~/.config/openshell/
clusters/<name>/mtls/"] + MTLS_DIR["~/.config/openshell/
gateways/<name>/mtls/"] end SERVER_CERT --> S1 @@ -49,9 +49,9 @@ graph TD ## Certificate Hierarchy -The PKI is a single-tier CA hierarchy generated by the `openshell-bootstrap` crate using `rcgen`. All certificates are created in a single pass at cluster bootstrap time. +The default PKI shape is a single-tier CA hierarchy. Operators can generate this bundle with their internal PKI tooling, cert-manager, or a local development CA. -``` +```text openshell-ca (Self-signed Root CA, O=openshell, CN=openshell-ca) ├── openshell-server (Leaf cert, CN=openshell-server) │ SANs: openshell, openshell.openshell.svc, @@ -60,28 +60,26 @@ openshell-ca (Self-signed Root CA, O=openshell, CN=openshell-ca) │ + extra SANs for remote deployments │ └── openshell-client (Leaf cert, CN=openshell-client) - Shared by the CLI and all sandbox pods. + Shared by the CLI and all sandbox workloads. ``` Key design decisions: -- **Single client certificate**: One client cert is shared by the CLI and every sandbox pod. This simplifies secret management. Individual sandbox identity is not expressed at the TLS layer; post-authentication identification uses the `x-sandbox-id` gRPC header. -- **Long-lived certificates**: Certificates use `rcgen` defaults (validity ~1975--4096), which effectively never expire. This is appropriate for an internal dev-cluster PKI where certificates are ephemeral to the cluster's lifetime. -- **CA key not persisted**: The CA private key is used only during generation and is not stored in any Kubernetes secret. Re-signing requires regenerating the entire PKI. - -See `crates/openshell-bootstrap/src/pki.rs:35` for the `generate_pki()` implementation and `crates/openshell-bootstrap/src/pki.rs:18` for the default SAN list. +- **Single client certificate**: One client cert is shared by the CLI and every sandbox workload. This simplifies secret management. Individual sandbox identity is not expressed at the TLS layer; post-authentication identification uses the `x-sandbox-id` gRPC header. +- **Certificate lifetime**: Certificate validity is owned by the operator's PKI policy. +- **CA key not stored in OpenShell**: The chart consumes certificates and CA bundles, but it does not need the CA private key. ## Kubernetes Secret Distribution -The PKI bundle is distributed as three Kubernetes secrets in the `openshell` namespace: +In Helm deployments, the PKI bundle is distributed as three Kubernetes secrets in the `openshell` namespace: | Secret Name | Type | Contents | Consumed By | |---|---|---|---| | `openshell-server-tls` | `kubernetes.io/tls` | `tls.crt` (server cert), `tls.key` (server key) | Gateway StatefulSet | | `openshell-server-client-ca` | `Opaque` | `ca.crt` (CA cert) | Gateway StatefulSet (client verification) | -| `openshell-client-tls` | `Opaque` | `tls.crt` (client cert), `tls.key` (client key), `ca.crt` (CA cert) | Sandbox pods, CLI (via local filesystem) | +| `openshell-client-tls` | `Opaque` | `tls.crt` (client cert), `tls.key` (client key), `ca.crt` (CA cert) | Sandbox workloads, CLI (via local filesystem) | -Secret names are defined as constants in `crates/openshell-bootstrap/src/constants.rs:6-10`. +Secret names are chart values under `server.tls.*` in `deploy/helm/openshell/values.yaml`. ### Gateway Mounts @@ -94,21 +92,21 @@ The Helm StatefulSet (`deploy/helm/openshell/templates/statefulset.yaml`) mounts Environment variables point the gateway binary to these paths: -``` +```text OPENSHELL_TLS_CERT=/etc/openshell-tls/server/tls.crt OPENSHELL_TLS_KEY=/etc/openshell-tls/server/tls.key OPENSHELL_TLS_CLIENT_CA=/etc/openshell-tls/client-ca/ca.crt ``` -### Sandbox Pod Mounts +### Sandbox Workload TLS Material -When the gateway creates a sandbox pod (`crates/openshell-server/src/sandbox/mod.rs:681`), it injects: +When the Kubernetes driver creates a sandbox pod, it injects: - A volume backed by the `openshell-client-tls` secret. - A read-only mount at `/etc/openshell-tls/client/` on the agent container. - Environment variables for the sandbox gRPC client: -``` +```text OPENSHELL_TLS_CA=/etc/openshell-tls/client/ca.crt OPENSHELL_TLS_CERT=/etc/openshell-tls/client/tls.crt OPENSHELL_TLS_KEY=/etc/openshell-tls/client/tls.key @@ -119,7 +117,7 @@ OPENSHELL_ENDPOINT=https://openshell.openshell.svc.cluster.local:8080 The CLI's copy of the client certificate bundle is written to: -``` +```text $XDG_CONFIG_HOME/openshell/gateways//mtls/ ├── ca.crt ├── tls.crt @@ -128,41 +126,31 @@ $XDG_CONFIG_HOME/openshell/gateways//mtls/ Files are written atomically using a temp-dir -> validate -> rename strategy with backup and rollback on failure. See `crates/openshell-bootstrap/src/mtls.rs:10`. -## PKI Bootstrap Sequence +## PKI Provisioning Sequence + +PKI provisioning is operator-driven: -PKI provisioning occurs during `deploy_cluster_with_logs()` (`crates/openshell-bootstrap/src/lib.rs:284`). The full sequence: +1. Generate or obtain a server certificate, server key, client certificate, client key, and CA certificate. +2. Provide the server certificate and client CA to the gateway process. +3. Provide the client certificate bundle to sandbox workloads through the selected compute driver. +4. Store the same client bundle under `~/.config/openshell/gateways//mtls/` so the CLI can authenticate to the gateway. -1. **Cluster container launched** -- a k3s container is created via Docker with a persistent volume. -2. **k3s readiness** -- the bootstrap waits for k3s to become ready inside the container. -3. **Extra SANs computed** -- for remote deployments, the SSH destination hostname and its resolved IP are added to the server certificate's SANs. For local deployments, the detected gateway host (if any) is added. -4. **`reconcile_pki()` called** (`crates/openshell-bootstrap/src/lib.rs:515`): - 1. Wait for the `openshell` namespace to exist (created by the Helm controller). - 2. Attempt to load existing PKI from the three K8s secrets via `kubectl get secret` exec'd inside the container. Each field is base64-decoded and validated for PEM markers. - 3. **If secrets exist and are valid**: reuse them and return `rotated=false`. - 4. **If secrets are missing, incomplete, or malformed**: generate fresh PKI via `generate_pki()`, apply all three secrets via `kubectl apply`, and return `rotated=true`. -5. **Workload restart on rotation** -- if `rotated=true` and the openshell StatefulSet already exists, the bootstrap performs `kubectl rollout restart` and waits for completion. This ensures the server picks up new TLS secrets before the CLI writes its local copy. -6. **CLI-side credential storage** -- `store_pki_bundle()` writes `ca.crt`, `tls.crt`, `tls.key` to the local filesystem. +For Helm deployments, steps 2 and 3 use the `openshell-server-tls`, `openshell-server-client-ca`, and `openshell-client-tls` Kubernetes secrets before installing or upgrading the chart. ```mermaid sequenceDiagram - participant CLI as nav deploy - participant Docker as Cluster Container - participant K8s as k3s / K8s API - - CLI->>Docker: Create container, wait for k3s - CLI->>K8s: Wait for openshell namespace - CLI->>K8s: Read existing TLS secrets - alt Secrets valid - CLI->>CLI: Reuse existing PKI - else Secrets missing/invalid - CLI->>CLI: generate_pki(extra_sans) - CLI->>K8s: kubectl apply (3 secrets) - alt Workload exists - CLI->>K8s: kubectl rollout restart - CLI->>K8s: Wait for rollout complete - end - end - CLI->>CLI: store_pki_bundle() to local filesystem + participant O as Operator + participant K8s as Kubernetes API + participant Helm as Helm + participant GW as Gateway Pod + participant CLI as CLI + + O->>K8s: Create TLS and SSH handshake secrets + O->>Helm: helm upgrade --install + Helm->>K8s: Apply StatefulSet and mounts + K8s->>GW: Start gateway with mounted certs + O->>CLI: Store client cert bundle locally + CLI->>GW: Connect with mTLS ``` ## Gateway TLS Enforcement @@ -183,7 +171,7 @@ The gateway supports three transport modes: ### Connection Flow -``` +```text TCP accept → TLS handshake (mandatory client cert in mTLS mode, optional in dual-auth mode) → hyper auto-negotiates HTTP/1.1 or HTTP/2 via ALPN @@ -216,7 +204,7 @@ The e2e test suite (`e2e/python/test_security_tls.py`) validates four scenarios: ## Sandbox-to-Gateway mTLS -Sandbox pods connect back to the gateway at startup to fetch their policy and provider credentials. The gRPC client (`crates/openshell-sandbox/src/grpc_client.rs:18`) reads three environment variables to configure mTLS: +Sandbox workloads connect back to the gateway at startup to fetch their policy and provider credentials. The gRPC client (`crates/openshell-sandbox/src/grpc_client.rs:18`) reads three environment variables to configure mTLS: | Env Var | Value | |---|---| @@ -225,10 +213,12 @@ Sandbox pods connect back to the gateway at startup to fetch their policy and pr | `OPENSHELL_TLS_KEY` | `/etc/openshell-tls/client/tls.key` | These are used to build a `tonic::transport::ClientTlsConfig` with: -- `ca_certificate()` -- verifies the server's certificate against the cluster CA. + +- `ca_certificate()` -- verifies the server's certificate against the deployment CA. - `identity()` -- presents the shared client certificate for mTLS. The sandbox calls two RPCs over this authenticated channel: + - `GetSandboxSettings` -- fetches the YAML policy that governs the sandbox's behavior. - `GetSandboxProviderEnvironment` -- fetches provider credentials as environment variables. @@ -269,34 +259,41 @@ The gateway enforces two concurrent connection limits to bound the impact of cre These limits are tracked in-memory and decremented when tunnels close. Exceeding either limit returns HTTP 429 (Too Many Requests). -### NSSH1 Handshake +### Supervisor-Initiated Relay Model -After the gateway connects to the sandbox pod's SSH port, it performs a cryptographic handshake: +The gateway never dials the sandbox. Instead, the sandbox supervisor opens an outbound `ConnectSupervisor` bidirectional gRPC stream to the gateway on startup and keeps it alive for the sandbox lifetime. SSH traffic for `/connect/ssh` (and exec traffic for `ExecSandbox`) rides this same TCP+TLS+HTTP/2 connection as separate multiplexed HTTP/2 streams. The gateway-side registry and `RelayStream` handler live in `crates/openshell-server/src/supervisor_session.rs`; the supervisor-side bridge lives in `crates/openshell-sandbox/src/supervisor_session.rs`. -``` -NSSH1 \n -``` +Per-connection flow: + +1. CLI presents `x-sandbox-id` + `x-sandbox-token` at `/connect/ssh` and passes gateway token validation. +2. Gateway calls `SupervisorSessionRegistry::open_relay(sandbox_id, ...)`, which allocates a `channel_id` (UUID) and sends a `RelayOpen` message to the supervisor over the already-established `ConnectSupervisor` stream. If no session is registered yet, it polls with exponential backoff up to a bounded timeout (30 s for `/connect/ssh`, 15 s for `ExecSandbox`). +3. The supervisor opens a new `RelayStream` RPC on the same `Channel` — a new HTTP/2 stream, no new TCP connection and no new TLS handshake. The first `RelayFrame` is a `RelayInit { channel_id }` that claims the pending slot on the gateway. +4. `claim_relay` pairs the gateway-side waiter with the supervisor-side RPC via a `tokio::io::duplex(64 KiB)` pair. Subsequent `RelayFrame::data` frames carry raw SSH bytes in both directions. The supervisor is a dumb byte bridge: it has no protocol awareness of the SSH bytes flowing through. +5. Inside the sandbox workload, the supervisor connects the relay to sshd over a Unix domain socket at `/run/openshell/ssh.sock`. + +Security properties of this model: -- **HMAC**: `HMAC-SHA256(secret, "{token}|{timestamp}|{nonce}")`, hex-encoded. -- **Secret**: shared via `OPENSHELL_SSH_HANDSHAKE_SECRET` env var, set on both the gateway and sandbox. -- **Clock skew tolerance**: configurable via `OPENSHELL_SSH_HANDSHAKE_SKEW_SECS` (default 300 seconds). -- **Expected response**: `OK\n` from the sandbox. +- **One auth boundary.** mTLS on the `ConnectSupervisor` stream is the only identity check between gateway and sandbox. Every relay rides that same authenticated HTTP/2 connection. +- **No inbound network path into the sandbox.** The sandbox exposes no TCP port for gateway ingress; all relays are supervisor-initiated. The workload only needs egress to the gateway. +- **In-workload access control is filesystem permissions on the Unix socket.** sshd listens on `/run/openshell/ssh.sock` with the parent directory at `0700` and the socket itself at `0600`, both owned by the supervisor (root). The sandbox entrypoint runs as an unprivileged user and cannot open either. Any process in the supervisor's filesystem view that can open the socket can reach sshd; this is the same trust model as any local Unix socket with `0600` permissions. See `crates/openshell-sandbox/src/ssh.rs:55-83`. +- **Supersede race is closed.** A supervisor reconnect registers a new `session_id` for the same sandbox id. Cleanup on the old session's task uses `remove_if_current(sandbox_id, session_id)` so a late-finishing old task cannot evict the new registration or serve relays meant for the new instance. See `SupervisorSessionRegistry::remove_if_current` in `crates/openshell-server/src/supervisor_session.rs`. +- **Pending-relay reaper.** A background task sweeps `pending_relays` entries older than 10 s (`RELAY_PENDING_TIMEOUT`). If the supervisor acknowledges `RelayOpen` but never initiates `RelayStream` — crash, deadlock, or adversarial stall — the gateway-side slot does not pin indefinitely. +- **Client-side keepalives.** The CLI's `ssh` invocation sets `ServerAliveInterval=15` / `ServerAliveCountMax=3` (`crates/openshell-cli/src/ssh.rs:150`), so a silently-dropped relay (gateway restart, supervisor restart, or adversarial TCP drop) surfaces to the user within roughly 45 s rather than hanging. -This handshake prevents direct connections to sandbox SSH ports from within the cluster, even from pods that share the network. +Observability (sandbox side, OCSF): `session_established`, `session_closed`, `session_failed`, `relay_open`, `relay_closed`, `relay_failed`, `relay_close_from_gateway` — all emitted as `NetworkActivity` events. Gateway-side OCSF emission for the same lifecycle is a tracked follow-up. ## Port Configuration -Traffic flows through several layers from the host to the gateway process: +Traffic flows through the configured gateway exposure path to the gateway process. Kubernetes deployments use the Helm-managed service; standalone deployments bind the gateway port directly or place it behind an operator-managed proxy. | Layer | Port | Configurable Via | |---|---|---| -| Host (Docker) | `8080` (default) | `--port` flag on `nav deploy` | -| Container | `30051` | Hardcoded in `crates/openshell-bootstrap/src/docker.rs` | -| k3s NodePort | `30051` | `deploy/helm/openshell/values.yaml` (`service.nodePort`) | -| k3s Service | `8080` | `deploy/helm/openshell/values.yaml` (`service.port`) | +| External ingress / port-forward / load balancer / reverse proxy | Operator choice | Platform-specific service or proxy configuration | +| Kubernetes Service | `8080` by default | `deploy/helm/openshell/values.yaml` (`service.port`) | +| NodePort, when enabled | `30051` by default | `deploy/helm/openshell/values.yaml` (`service.nodePort`) | | Server bind | `8080` | `--port` flag / `OPENSHELL_SERVER_PORT` env var | -Docker maps `host_port → 30051/tcp`. Inside k3s, the NodePort service maps `30051 → 8080 (pod port)`. The server binds `0.0.0.0:8080`. +The server binds `0.0.0.0:8080` by default. The chart maps the service port to the gateway workload's `grpc` port for Kubernetes deployments. ## Security Model Summary @@ -314,19 +311,19 @@ graph LR API["gRPC + HTTP API"] end - subgraph KUBE["Kubernetes"] - SBX["Sandbox Pod"] + subgraph PLATFORM["Compute Platform"] + SBX["Sandbox Workload"] end subgraph INET["Internet"] HOSTS["Allowed Hosts"] end - CLI -- "mTLS
(cluster CA)" --> TLS - SDK -- "mTLS
(cluster CA)" --> TLS + CLI -- "mTLS
(deployment CA)" --> TLS + SDK -- "mTLS
(deployment CA)" --> TLS TLS --> API - SBX -- "mTLS
(cluster CA)" --> TLS - API -- "SSH + NSSH1
handshake" --> SBX + SBX -- "mTLS + ConnectSupervisor
(supervisor-initiated)" --> TLS + API -- "RelayStream
(HTTP/2 on same mTLS conn)" --> SBX SBX -- "OPA policy +
process identity" --> HOSTS ``` @@ -334,9 +331,10 @@ graph LR | Boundary | Mechanism | |---|---| -| External → Gateway | mTLS with cluster CA by default, or trusted reverse-proxy/Cloudflare boundary in edge mode | -| Sandbox → Gateway | mTLS with shared client cert | -| Gateway → Sandbox (SSH) | Session token + HMAC-SHA256 handshake (NSSH1) | +| External → Gateway | mTLS with deployment CA by default, or trusted reverse-proxy/Cloudflare boundary in edge mode | +| Sandbox → Gateway | mTLS with shared client cert (supervisor-initiated `ConnectSupervisor` stream) | +| Gateway → Sandbox (SSH/exec) | Rides the supervisor's mTLS `ConnectSupervisor` HTTP/2 connection as a `RelayStream`; no separate gateway-to-sandbox network connection | +| Supervisor → workload sshd | Unix-socket filesystem permissions (`/run/openshell/ssh.sock`, 0700 parent / 0600 socket) | | Sandbox → External (network) | OPA policy + process identity binding via `/proc` | ### What Is Not Authenticated (by Design) @@ -346,7 +344,7 @@ graph LR ### Gateway Security Context -The gateway container runs with a hardened security context (`deploy/helm/openshell/values.yaml:25`): +The gateway workload runs with a hardened security context (`deploy/helm/openshell/values.yaml:25`): ```yaml securityContext: @@ -377,18 +375,21 @@ This section defines the primary attacker profiles, what the current design prot |---|---| | Network attacker | Can observe/modify traffic between clients and gateway | | Unauthorized external client | Can reach gateway port but has no valid client cert | -| Compromised sandbox workload | Has code execution inside one sandbox pod | -| Malicious in-cluster pod | Can attempt direct pod-to-pod connections | +| Compromised sandbox workload | Has code execution inside one sandbox workload | +| Malicious platform peer | Can attempt direct workload-to-workload connections | | Stolen CLI credentials | Has copied `ca.crt`/`tls.crt`/`tls.key` from a developer machine | ### Primary Defenses | Threat | Existing Defense | Notes | |---|---|---| -| MITM or passive interception of gateway traffic | Mandatory mTLS with cluster CA, or trusted reverse-proxy boundary in Cloudflare mode | Default mode is direct mTLS; reverse-proxy mode shifts the outer trust boundary upstream | +| MITM or passive interception of gateway traffic | Mandatory mTLS with deployment CA, or trusted reverse-proxy boundary in Cloudflare mode | Default mode is direct mTLS; reverse-proxy mode shifts the outer trust boundary upstream | | Unauthenticated API/health access | mTLS by default, or Cloudflare/reverse-proxy auth in edge mode | `/health*` are direct-mTLS only in the default deployment mode | -| Forged SSH tunnel connection to sandbox | Session token validation + NSSH1 HMAC handshake | Requires token and shared handshake secret | -| Direct access to sandbox SSH port from cluster peers | NSSH1 challenge-response | Connection denied without valid signature | +| Forged SSH tunnel connection to sandbox | Session token validation at the gateway; only the supervisor's authenticated mTLS `ConnectSupervisor` stream can carry a `RelayStream` to its sandbox | Forging a relay requires stealing a valid mTLS client identity | +| Direct access to sandbox sshd from platform peers | sshd listens on a Unix socket (`0700` parent / `0600` socket) inside the workload | No network path exists to sshd from platform peers | +| Stale or reconnecting supervisor serves relays for a new instance | `session_id`-scoped `remove_if_current` on the registry | Old session cleanup cannot evict a newer registration | +| Supervisor acknowledges `RelayOpen` but never initiates `RelayStream` | Gateway-side pending-relay reaper (10 s timeout) | Prevents indefinite resource pinning by a buggy or malicious supervisor | +| Silent TCP drop of an in-flight relay | CLI `ServerAliveInterval=15` / `ServerAliveCountMax=3` | Client detects a dead relay within ~45 s instead of hanging | | Unauthorized outbound internet access from sandbox | OPA policy + process identity checks | Applies to sandbox egress policy layer | ### Residual Risks and Current Tradeoffs @@ -404,35 +405,35 @@ This section defines the primary attacker profiles, what the current design prot ### Out of Scope / Not Defended By This Layer -- A fully compromised Kubernetes control plane or cluster-admin account. -- A malicious actor with direct access to Kubernetes secrets in the `openshell` namespace. +- A fully compromised compute platform, such as a Kubernetes control plane, container host, or VM host. +- A malicious actor with direct access to deployment secrets for the gateway or sandbox runtime. - Host-level compromise of the developer workstation running the CLI. - Application-layer authorization bugs after mTLS authentication succeeds. ### Trust Assumptions -- The cluster CA is generated and distributed without interception during bootstrap. -- Kubernetes secret access is restricted to intended workloads and operators. +- The deployment CA is generated and distributed without interception during provisioning. +- Secret access is restricted to intended workloads and operators. - Gateway and sandbox container images are trusted and not tampered with. -- System clocks are reasonably synchronized for timestamp-based SSH handshake checks. +- The sandbox workload's filesystem is trusted: only the supervisor process (root) can open `/run/openshell/ssh.sock`, which is enforced by the `0700` parent directory and `0600` socket permissions set at sshd start. ## Sandbox Outbound TLS (L7 Inspection) -Separate from the cluster mTLS infrastructure, each sandbox has an independent TLS capability for inspecting outbound HTTPS traffic. This is documented here for completeness because it involves a distinct, per-sandbox PKI. +Separate from the gateway mTLS infrastructure, each sandbox has an independent TLS capability for inspecting outbound HTTPS traffic. This is documented here for completeness because it involves a distinct, per-sandbox PKI. The sandbox proxy automatically detects and terminates TLS on outbound HTTPS connections by peeking the first bytes of each tunnel. This enables credential injection and L7 inspection without requiring explicit policy configuration. The proxy performs TLS man-in-the-middle inspection: -1. **Ephemeral sandbox CA**: a per-sandbox CA (`CN=OpenShell Sandbox CA, O=OpenShell`) is generated at sandbox startup. This CA is completely independent of the cluster mTLS CA. +1. **Ephemeral sandbox CA**: a per-sandbox CA (`CN=OpenShell Sandbox CA, O=OpenShell`) is generated at sandbox startup. This CA is completely independent of the gateway mTLS CA. 2. **Trust injection**: the sandbox CA is written to the sandbox filesystem and injected via `NODE_EXTRA_CA_CERTS` and `SSL_CERT_FILE` so processes inside the sandbox trust it. 3. **Dynamic leaf certs**: for each target hostname, the proxy generates and caches a leaf certificate signed by the sandbox CA (up to 256 entries). -4. **Upstream verification**: the proxy verifies upstream server certificates against Mozilla root CAs (`webpki-roots`), not against the cluster CA. +4. **Upstream verification**: the proxy verifies upstream server certificates against Mozilla root CAs (`webpki-roots`) and system CA certificates from the container's trust store, not against the gateway mTLS CA. Custom sandbox images can add corporate/internal CAs via `update-ca-certificates`. This capability is orthogonal to gateway mTLS -- it operates only on sandbox-to-internet traffic and uses entirely separate key material. See [Policy Language](security-policy.md) for configuration details. ## Cross-References - [Gateway Architecture](gateway.md) -- protocol multiplexing, gRPC services, persistence, and SSH tunneling -- [Cluster Bootstrap](cluster-single-node.md) -- cluster provisioning, Docker container lifecycle, and credential management +- [Gateway Deployment and Compute Platforms](gateway-single-node.md) -- gateway deployment modes, compute platform inputs, and removed k3s responsibilities - [Sandbox Architecture](sandbox.md) -- sandbox-side isolation, proxy, and policy enforcement - [Sandbox Connect](sandbox-connect.md) -- client-side SSH connection flow through the gateway - [Policy Language](security-policy.md) -- YAML/Rego policy system including L7 TLS inspection configuration diff --git a/architecture/gateway-settings.md b/architecture/gateway-settings.md index ef9538f5b0..0b6da9bdd8 100644 --- a/architecture/gateway-settings.md +++ b/architecture/gateway-settings.md @@ -30,9 +30,8 @@ The `REGISTERED_SETTINGS` static array defines the allowed setting keys and thei ```rust pub const REGISTERED_SETTINGS: &[RegisteredSetting] = &[ - RegisteredSetting { key: "log_level", kind: SettingValueKind::String }, - RegisteredSetting { key: "dummy_int", kind: SettingValueKind::Int }, - RegisteredSetting { key: "dummy_bool", kind: SettingValueKind::Bool }, + RegisteredSetting { key: "providers_v2_enabled", kind: SettingValueKind::Bool }, + RegisteredSetting { key: "ocsf_json_enabled", kind: SettingValueKind::Bool }, ]; ``` @@ -45,6 +44,7 @@ pub const REGISTERED_SETTINGS: &[RegisteredSetting] = &[ The reserved key `policy` is excluded from the registry. It is handled by dedicated policy commands and stored as a hex-encoded protobuf `SandboxPolicy` in the global settings' `Bytes` variant. Attempts to set or delete the `policy` key through settings commands are rejected. Helper functions: + - `setting_for_key(key)` -- look up a `RegisteredSetting` by name, returns `None` for unknown keys - `registered_keys_csv()` -- comma-separated list of valid keys for error messages - `parse_bool_like(raw)` -- flexible bool parsing from CLI string input @@ -83,6 +83,7 @@ The `UpdateSettings` RPC multiplexes policy and setting mutations through a sing | `global` | `bool` | Target gateway-global scope instead of sandbox scope | Validation rules: + - `policy` and `setting_key` cannot both be present - At least one of `policy` or `setting_key` must be present - `delete_setting` cannot be combined with a `policy` payload @@ -266,7 +267,7 @@ This prevents conflicting values at different scopes. An operator must delete a When a global policy is set, sandbox-scoped policy updates via `UpdateSettings` are rejected with `FailedPrecondition`: -``` +```text policy is managed globally; delete global policy before sandbox policy update ``` @@ -371,15 +372,14 @@ Set a single setting key at sandbox or global scope. ```bash # Sandbox-scoped -openshell settings set my-sandbox --key log_level --value debug +openshell settings set my-sandbox --key ocsf_json_enabled --value true # Global (requires confirmation) -openshell settings set --global --key log_level --value warn -openshell settings set --global --key dummy_bool --value yes -openshell settings set --global --key dummy_int --value 42 +openshell settings set --global --key providers_v2_enabled --value true +openshell settings set --global --key ocsf_json_enabled --value true # Skip confirmation -openshell settings set --global --key log_level --value info --yes +openshell settings set --global --key providers_v2_enabled --value true --yes ``` Value parsing is type-aware: bool keys accept `true/false/yes/no/1/0/on/off` via `parse_bool_like()`. Int keys parse as base-10 `i64`. String keys accept any value. @@ -390,7 +390,7 @@ Delete a setting key from the specified scope. ```bash # Global delete (unlocks sandbox control) -openshell settings delete --global --key log_level --yes +openshell settings delete --global --key providers_v2_enabled --yes ``` ### `policy set --global --policy FILE [--yes]` @@ -442,6 +442,7 @@ openshell policy get --global --full All `--global` mutations require human-in-the-loop confirmation via an interactive prompt. The `--yes` flag bypasses the prompt for scripted/CI usage. In non-interactive mode (no TTY), `--yes` is required -- otherwise the command fails with an error. The confirmation message varies: + - **Global setting set**: warns that this will override sandbox-level values for the key - **Global setting delete**: warns that this re-enables sandbox-level management - **Global policy set**: warns that this overrides all sandbox policies @@ -499,26 +500,26 @@ Settings are refreshed on each 2-second polling tick alongside the sandbox list ## Data Flow: Setting a Global Key -End-to-end trace for `openshell settings set --global --key log_level --value debug --yes`: +End-to-end trace for `openshell settings set --global --key providers_v2_enabled --value true --yes`: 1. **CLI** (`crates/openshell-cli/src/run.rs` -- `gateway_setting_set()`): - - `parse_cli_setting_value("log_level", "debug")` -- looks up `SettingValueKind::String` in the registry, wraps as `SettingValue { string_value: "debug" }` + - `parse_cli_setting_value("providers_v2_enabled", "true")` -- looks up `SettingValueKind::Bool` in the registry, wraps as `SettingValue { bool_value: true }` - `confirm_global_setting_takeover()` -- skipped because `--yes` - - Sends `UpdateSettingsRequest { setting_key: "log_level", setting_value: Some(...), global: true }` + - Sends `UpdateSettingsRequest { setting_key: "providers_v2_enabled", setting_value: Some(...), global: true }` 2. **Gateway** (`crates/openshell-server/src/grpc.rs` -- `update_settings()`): - Acquires `settings_mutex` for the duration of the operation - Detects `global=true`, `has_setting=true` - - `validate_registered_setting_key("log_level")` -- passes (key is in registry) + - `validate_registered_setting_key("providers_v2_enabled")` -- passes (key is in registry) - `load_global_settings()` -- reads `gateway_settings` record from store - - `proto_setting_to_stored()` -- converts proto value to `StoredSettingValue::String("debug")` + - `proto_setting_to_stored()` -- converts proto value to `StoredSettingValue::Bool(true)` - `upsert_setting_value()` -- inserts into `BTreeMap`, returns `true` (changed) - Increments `revision`, calls `save_global_settings()` - Returns `UpdateSettingsResponse { settings_revision: N }` 3. **Sandbox** (next poll tick in `run_policy_poll_loop()`): - `poll_settings(sandbox_id)` returns new `config_revision` - - `log_setting_changes()` logs: `Setting changed key="log_level" old="" new="debug"` + - `log_setting_changes()` logs: `Setting changed key="providers_v2_enabled" old="" new="true"` - `policy_hash` unchanged -- no OPA reload - Updates tracked `current_config_revision` and `current_settings` diff --git a/architecture/gateway-single-node.md b/architecture/gateway-single-node.md index 6389c728e1..4b5a6aa6d1 100644 --- a/architecture/gateway-single-node.md +++ b/architecture/gateway-single-node.md @@ -1,469 +1,129 @@ -# Gateway Bootstrap Architecture +# Gateway Deployment and Compute Platforms -This document describes how OpenShell bootstraps a single-node k3s gateway inside a Docker container, for both local and remote (SSH) targets. +This document describes the OpenShell gateway deployment model. Operators run a gateway endpoint and configure the compute driver that should create sandboxes. + +The Helm chart remains in this repository as the supported Kubernetes deployment artifact. Docker, Podman, and the experimental MicroVM runtime remain first-class compute platforms for local and specialized deployments. ## Goals and Scope -- Provide a single bootstrap flow through `openshell-bootstrap` for local and remote gateway lifecycle. -- Keep Docker as the only runtime dependency for provisioning and lifecycle operations. -- Package the OpenShell gateway as one container image, transferred to the target host via registry pull. -- Support idempotent `deploy` behavior (safe to re-run). -- Persist gateway access artifacts (metadata, mTLS certs) in the local XDG config directory. -- Track the active gateway so most CLI commands resolve their target automatically. +- Keep the gateway deployable as a standard process, container, or Kubernetes Helm release. +- Keep the Helm chart for Kubernetes deployments. +- Keep the gateway image independent from the compute runtime. +- Make compute-platform dependencies explicit. +- Preserve CLI gateway registration and selection as the way users target an already-running gateway. Out of scope: -- Multi-node orchestration. +- Provisioning Kubernetes, Docker, Podman, or VM host infrastructure. +- Defining a new one-command mTLS import flow for every deployment type. ## Components -- `crates/openshell-cli/src/main.rs`: CLI entry point; `clap`-based command parsing. -- `crates/openshell-cli/src/run.rs`: CLI command implementations (`gateway_start`, `gateway_stop`, `gateway_destroy`, `gateway_info`, `doctor_logs`). -- `crates/openshell-cli/src/bootstrap.rs`: Auto-bootstrap helpers for `sandbox create` (offers to deploy a gateway when one is unreachable). -- `crates/openshell-bootstrap/src/lib.rs`: Gateway lifecycle orchestration (`deploy_gateway`, `deploy_gateway_with_logs`, `gateway_handle`, `check_existing_deployment`). -- `crates/openshell-bootstrap/src/docker.rs`: Docker API wrappers (per-gateway network, volume, container, image operations). -- `crates/openshell-bootstrap/src/image.rs`: Remote image registry pull with XOR-obfuscated distribution credentials. -- `crates/openshell-bootstrap/src/runtime.rs`: In-container operations via `docker exec` (health polling, stale node cleanup, deployment restart). -- `crates/openshell-bootstrap/src/metadata.rs`: Gateway metadata creation, storage, and active gateway tracking. -- `crates/openshell-bootstrap/src/mtls.rs`: Gateway TLS detection and CLI mTLS bundle extraction. -- `crates/openshell-bootstrap/src/push.rs`: Local development image push into k3s containerd. -- `crates/openshell-bootstrap/src/paths.rs`: XDG path resolution. -- `crates/openshell-bootstrap/src/constants.rs`: Shared constants (image name, container/volume/network naming). -- `deploy/docker/Dockerfile.images` (target `cluster`): Container image definition (k3s base + Helm charts + manifests + entrypoint). -- `deploy/docker/cluster-entrypoint.sh`: Container entrypoint (DNS proxy, registry config, manifest injection). -- `deploy/docker/cluster-healthcheck.sh`: Docker HEALTHCHECK script. -- Docker daemon(s): - - Local daemon for local deploys. - - Remote daemon over SSH for remote deploy container operations. - -## CLI Commands - -All gateway lifecycle commands live under `openshell gateway`: - -| Command | Description | -|---|---| -| `openshell gateway start [--name NAME] [--remote user@host] [--ssh-key PATH]` | Provision or update a gateway | -| `openshell gateway stop [--name NAME] [--remote user@host]` | Stop the container (preserves state) | -| `openshell gateway destroy [--name NAME] [--remote user@host]` | Destroy container, attached volumes, per-gateway network, and metadata | -| `openshell gateway info [--name NAME]` | Show deployment details (endpoint, SSH host) | -| `openshell status` | Show gateway health via gRPC/HTTP | -| `openshell doctor logs [--name NAME] [--remote user@host] [--tail N]` | Fetch gateway container logs | -| `openshell doctor exec [--name NAME] [--remote user@host] -- ` | Run a command inside the gateway container | -| `openshell gateway select ` | Set the active gateway | -| `openshell gateway select` | Open an interactive chooser on a TTY, or list all gateways in non-interactive mode | - -The `--name` flag defaults to `"openshell"`. When omitted on commands that accept it, the CLI resolves the active gateway via: `--gateway` flag, then `OPENSHELL_GATEWAY` env, then `~/.config/openshell/active_gateway` file. - -For remote dev/test deploys from a local checkout, `scripts/remote-deploy.sh` -wraps a different workflow: it rsyncs the repository to a remote host, builds -the release CLI plus cluster/server/sandbox images on that machine, and then -invokes `openshell gateway start` with explicit flags such as `--recreate`, -`--plaintext`, or `--disable-gateway-auth` only when requested. +- `crates/openshell-server`: Gateway API server, persistence, inference route management, SSH relay, and compute-driver integration. +- `crates/openshell-driver-kubernetes`: Kubernetes compute driver for sandbox pods and Kubernetes resources. +- `crates/openshell-driver-docker`: Docker compute driver for local sandbox containers. +- `crates/openshell-driver-vm`: VM compute driver for libkrun-backed sandboxes. +- Podman driver path: rootless container execution compatible with the Podman runtime model. +- `deploy/helm/openshell`: Helm chart for deploying the gateway and Kubernetes driver configuration. +- `deploy/docker/Dockerfile.images` target `gateway`: Builds the published gateway image. +- `crates/openshell-cli`: CLI commands that register, select, and talk to gateways. -## Local Task Flows (`mise`) - -Development task entrypoints split bootstrap behavior: - -| Task | Behavior | -|---|---| -| `mise run cluster` | Bootstrap or incremental deploy: creates gateway if needed (fast recreate), then detects changed files and rebuilds/pushes only impacted components | - -For `mise run cluster`, `.env` acts as local source-of-truth for `GATEWAY_NAME`, `GATEWAY_PORT`, and `OPENSHELL_GATEWAY`. Missing keys are appended; existing values are preserved. If `GATEWAY_PORT` is missing, the task selects a free local port and persists it. -Fast mode ensures a local registry (`127.0.0.1:5000`) is running and configures k3s to mirror pulls via `host.docker.internal:5000`, so the cluster task can push/pull local component images consistently. - -## Bootstrap Sequence Diagram +## Deployment Flow ```mermaid sequenceDiagram - participant U as User - participant C as openshell-cli - participant B as openshell-bootstrap - participant L as Local Docker daemon - participant R as Remote Docker daemon (SSH) - - U->>C: openshell gateway start --remote user@host - C->>B: deploy_gateway(DeployOptions) - - B->>B: create_ssh_docker_client (ssh://, 600s timeout) - B->>R: pull_remote_image (registry auth, platform-aware) - B->>R: tag image to local ref - - Note over B,R: Docker socket APIs only, no extra host dependencies - - B->>B: resolve SSH host for extra TLS SANs - B->>R: ensure_network (per-gateway bridge, attachable) - B->>R: ensure_volume - B->>R: ensure_container (privileged, k3s server) - B->>R: start_container - B->>R: clean_stale_nodes (kubectl delete node) - B->>R: wait_for_gateway_ready (180 attempts, 2s apart) - B->>R: poll for secret openshell-cli-client (90 attempts, 2s apart) - R-->>B: ca.crt, tls.crt, tls.key - B->>B: atomically store mTLS bundle - B->>B: create and persist gateway metadata JSON - B->>B: save_active_gateway - B-->>C: GatewayHandle (metadata, docker client) -``` - -## End-State Connectivity Diagram - -```mermaid -flowchart LR - subgraph WS[User workstation] - NAV[openshell-cli] - MTLS[mTLS bundle ca.crt, tls.crt, tls.key] - end - - subgraph HOST[Target host] - DOCKER[Docker daemon] - K3S[openshell-cluster-NAME single k3s container] - G8080[Gateway :port -> :30051 mTLS default 8080] - SBX[Sandbox runtime] - end - - NAV --> G8080 - MTLS -. client cert auth .-> G8080 - - DOCKER --> K3S - K3S --> G8080 - K3S --> G80 - K3S --> G443 + participant O as Operator + participant G as Gateway + participant D as Compute Driver + participant P as Compute Platform + participant C as openshell CLI + + O->>G: Start gateway with driver configuration + G->>D: Initialize selected compute driver + D->>P: Verify runtime or cluster access + O->>C: Register reachable gateway endpoint + C->>G: gRPC / HTTP requests + G->>D: Create / delete / watch sandboxes + D->>P: Create sandbox workload ``` -## Deploy Flow - -### 1) Entry and client selection - -`deploy_gateway(DeployOptions)` in `crates/openshell-bootstrap/src/lib.rs` chooses execution mode: - -- `DeployOptions` fields: `name: String`, `image_ref: Option`, `remote: Option`, `port: u16` (default 8080). -- `RemoteOptions` fields: `destination: String`, `ssh_key: Option`. -- **Local deploy**: Create one Docker client with `Docker::connect_with_local_defaults()`. -- **Remote deploy**: Create SSH Docker client via `Docker::connect_with_ssh()` with a 600-second timeout (for large image transfers). The destination is prefixed with `ssh://` if not already present. - -The `deploy_gateway_with_logs` variant accepts an `FnMut(String)` callback for progress reporting. The CLI wraps this in a `GatewayDeployLogPanel` for interactive terminals. +## Supported Compute Platforms -**Pre-deploy check** (CLI layer in `gateway_start`): In interactive terminals, `check_existing_deployment` inspects whether a container or volume already exists. If found, the user is prompted to destroy and recreate or reuse the existing gateway. +| Platform | Gateway shape | Sandbox workload | Primary dependencies | +|---|---|---|---| +| Docker | Standalone gateway process or container on a host with Docker access. | Local containers. | Docker daemon, image pull/build access, local networking. | +| Podman | Standalone gateway process with Podman socket access. | Rootless or user-scoped containers. | Podman socket, rootless networking, image pull/build access. | +| Kubernetes | Gateway StatefulSet installed by Helm. | Sandbox pods. | Kubernetes API, namespace, service account, RBAC, storage, secrets. | +| MicroVM | Gateway process with VM driver access. | VM-backed sandboxes. | VM runtime rootfs, libkrun-based driver, host virtualization support. | -### 2) Image readiness +## Kubernetes Helm Deployment -Image ref resolution in `default_gateway_image_ref()`: - -1. If `OPENSHELL_CLUSTER_IMAGE` is set and non-empty, use it verbatim. -2. Otherwise, use the published distribution image base (`/openshell/cluster`) with its default tag behavior. - -- **Local deploy**: `ensure_image()` inspects the image on the local daemon and pulls from the configured registry if missing (using built-in distribution credentials when pulling from the default distribution host). -- **Remote deploy**: `pull_remote_image()` queries the remote daemon's architecture via `Docker::version()`, pulls the matching platform variant from the distribution registry (with XOR-decoded credentials), and tags the pulled image to the expected local ref (for example `openshell/cluster:dev` when an explicit local tag is requested). - -### 3) Runtime infrastructure - -For the target daemon (local or remote): - -1. **Ensure bridge network** `openshell-cluster-{name}` (attachable, bridge driver) via `ensure_network()`. Each gateway gets its own isolated Docker network. -2. **Ensure volume** `openshell-cluster-{name}` via `ensure_volume()`. -3. **Compute extra TLS SANs**: - - For **local deploys**: Check `DOCKER_HOST` for a non-loopback `tcp://` endpoint (e.g., `tcp://docker:2375` in CI). If found, extract the host as an extra SAN. The function `local_gateway_host_from_docker_host()` skips `localhost`, `127.0.0.1`, and `::1`. - - For **remote deploys**: Extract the host from the SSH destination (handles `user@host`, `ssh://user@host`), resolve via `ssh -G` to get the canonical hostname/IP. Include both the resolved host and original SSH host (if different) as extra SANs. -4. **Ensure container** `openshell-cluster-{name}` via `ensure_container()`: - - k3s server command: `server --disable=traefik --tls-san=127.0.0.1 --tls-san=localhost --tls-san=host.docker.internal` plus computed extra SANs. - - Privileged mode. - - Volume bind mount: `openshell-cluster-{name}:/var/lib/rancher/k3s`. - - Network: `openshell-cluster-{name}` (per-gateway bridge network). - - Extra host: `host.docker.internal:host-gateway`. - - The cluster entrypoint prefers the resolved IPv4 for `host.docker.internal` when populating sandbox pod `hostAliases`, then falls back to the container default gateway. This keeps sandbox host aliases working on Docker Desktop, where the host-reachable IP differs from the bridge gateway. - - Port mappings: - - | Container Port | Host Port | Purpose | - |---|---|---| - | 30051/tcp | configurable (default 8080) | OpenShell service NodePort (mTLS) | - - - Container environment variables (see [Container Environment Variables](#container-environment-variables) below). - - If the container exists with a different image ID (compared by inspecting the content-addressable ID), it is stopped, force-removed, and recreated. If the image matches, the existing container is reused. -5. **Start container** via `start_container()`. Tolerates already-running 409 conflict. - -### 4) Readiness and artifact extraction - -After the container starts: - -1. **Clean stale nodes**: `clean_stale_nodes()` finds nodes whose name does not match the deterministic k3s `--node-name` and deletes them. That node name is derived from the gateway name but normalized to a Kubernetes-safe lowercase form so existing gateway names that contain `_`, `.`, or uppercase characters still produce a valid node identity. This cleanup is needed when a container is recreated but reuses the persistent volume -- old node entries can persist in etcd. Non-fatal on error; returns the count of removed nodes. -2. **Push local images** (optional, local deploy only): If `OPENSHELL_PUSH_IMAGES` is set, the comma-separated image refs are exported from the local Docker daemon as a single tar, uploaded into the container via `docker put_archive`, and imported into containerd via `ctr images import` in the `k8s.io` namespace. After import, `kubectl rollout restart deployment/openshell openshell` is run, followed by `kubectl rollout status --timeout=180s` to wait for completion. See `crates/openshell-bootstrap/src/push.rs`. -3. **Wait for gateway health**: `wait_for_gateway_ready()` polls the Docker HEALTHCHECK status up to 180 times, 2 seconds apart (6 min total). A background task streams container logs during this wait. Failure modes: - - Container exits during polling: error includes recent log lines. - - Container has no HEALTHCHECK instruction: fails immediately. - - HEALTHCHECK reports unhealthy on final attempt: error includes recent logs. - -The gateway StatefulSet also uses a Kubernetes `startupProbe` on the gRPC port before steady-state liveness and readiness checks begin. This gives single-node k3s boots extra time to absorb early networking and flannel initialization delay without restarting the gateway pod too aggressively. - -### 5) mTLS bundle capture - -TLS is always required. `fetch_and_store_cli_mtls()` polls for Kubernetes secret `openshell-cli-client` in namespace `openshell` (90 attempts, 2 seconds apart, 3 min total). Each attempt checks the container is still running. The secret's base64-encoded `ca.crt`, `tls.crt`, and `tls.key` fields are decoded and stored. - -Storage location: `~/.config/openshell/gateways/{name}/mtls/` - -Write is atomic: write to `.tmp` directory, validate all three files are non-empty, rename existing directory to `.bak`, rename `.tmp` to final path, then remove `.bak`. - -### 6) Metadata persistence - -`create_gateway_metadata()` produces a `GatewayMetadata` struct: - -- **Local**: endpoint `https://127.0.0.1:{port}` by default, or `https://{docker_host}:{port}` when `DOCKER_HOST` is a non-loopback `tcp://` endpoint. `is_remote=false`. -- **Remote**: endpoint `https://{resolved_host}:{port}`, `is_remote=true`, plus SSH destination and resolved host. - -Metadata fields: - -| Field | Type | Description | -|---|---|---| -| `name` | `String` | Gateway name | -| `gateway_endpoint` | `String` | HTTPS endpoint with port (e.g., `https://127.0.0.1:8080`) | -| `is_remote` | `bool` | Whether gateway is remote | -| `gateway_port` | `u16` | Host port mapped to the gateway NodePort | -| `remote_host` | `Option` | SSH destination (e.g., `user@host`) | -| `resolved_host` | `Option` | Resolved hostname/IP from `ssh -G` | - -Metadata location: `~/.config/openshell/gateways/{name}_metadata.json` - -Note: metadata is stored at the `gateways/` level (not nested inside `{name}/` like mTLS). - -After deploy, the CLI calls `save_active_gateway(name)`, writing the gateway name to `~/.config/openshell/active_gateway`. Subsequent commands that don't specify `--gateway` or `OPENSHELL_GATEWAY` resolve to this active gateway. - -## Container Image - -The cluster image is defined by target `cluster` in `deploy/docker/Dockerfile.images`: - -``` -Base: rancher/k3s:v1.35.2-k3s1 -``` +The Helm chart at `deploy/helm/openshell` owns Kubernetes deployment concerns: -Layers added: +- Gateway StatefulSet and persistent volume claim. +- Service account, RBAC, and service. +- Gateway service exposure. +- TLS secret mounts and environment variables. +- Sandbox namespace, default sandbox image, and callback endpoint configuration. +- NetworkPolicy restricting sandbox SSH ingress to the gateway. -1. Custom entrypoint: `deploy/docker/cluster-entrypoint.sh` -> `/usr/local/bin/cluster-entrypoint.sh` -2. Healthcheck script: `deploy/docker/cluster-healthcheck.sh` -> `/usr/local/bin/cluster-healthcheck.sh` -3. Packaged Helm charts: `deploy/docker/.build/charts/*.tgz` -> `/var/lib/rancher/k3s/server/static/charts/` -4. Kubernetes manifests: `deploy/kube/manifests/*.yaml` -> `/opt/openshell/manifests/` +The chart expects these operator-provided inputs: -Bundled manifests include: -- `openshell-helmchart.yaml` (OpenShell Helm chart auto-deploy) -- `envoy-gateway-helmchart.yaml` (Envoy Gateway for Gateway API) -- `agent-sandbox.yaml` - -The HEALTHCHECK is configured as: `--interval=5s --timeout=5s --start-period=20s --retries=60`. - -## Entrypoint Script - -`deploy/docker/cluster-entrypoint.sh` runs before k3s starts. It performs: - -### DNS proxy setup - -On Docker custom networks, `/etc/resolv.conf` contains `127.0.0.11` (Docker's internal DNS). k3s detects this loopback and falls back to `8.8.8.8`, which does not work on Docker Desktop. The entrypoint solves this by: - -1. Discovering Docker's real DNS listener ports from the `DOCKER_OUTPUT` iptables chain. -2. Getting the container's `eth0` IP as a routable address. -3. Adding DNAT rules in PREROUTING to forward DNS from pod namespaces through to Docker's DNS. -4. Writing a custom resolv.conf pointing to the container IP. -5. Passing `--kubelet-arg=resolv-conf=/etc/rancher/k3s/resolv.conf` to k3s. - -Falls back to `8.8.8.8` / `8.8.4.4` if iptables detection fails. - -### Registry configuration - -Writes `/etc/rancher/k3s/registries.yaml` from `REGISTRY_HOST`, `REGISTRY_ENDPOINT`, `REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, and `REGISTRY_INSECURE` environment variables so that k3s/containerd can authenticate when pulling component images at runtime. When no explicit credentials are provided (the default for public GHCR repos), the auth block is omitted and images are pulled anonymously. - -### Manifest injection - -Copies bundled manifests from `/opt/openshell/manifests/` to `/var/lib/rancher/k3s/server/manifests/`. This is needed because the volume mount on `/var/lib/rancher/k3s` overwrites any files baked into that path at image build time. - -### Image configuration overrides - -When environment variables are set, the entrypoint modifies the HelmChart manifest at `/var/lib/rancher/k3s/server/manifests/openshell-helmchart.yaml`: - -- `IMAGE_REPO_BASE`: Rewrites `repository:`, `sandboxImage:`, and `jobImage:` in the HelmChart. -- `PUSH_IMAGE_REFS`: In push mode, parses comma-separated image refs and rewrites the exact gateway, sandbox, and pki-job image references (matching on path component `/gateway:`, `/sandbox:`, `/pki-job:`). -- `IMAGE_TAG`: Replaces `:latest` tags with the specified tag on gateway, sandbox, and pki-job images. Handles both quoted and unquoted `tag: latest` formats. -- `IMAGE_PULL_POLICY`: Replaces `pullPolicy: Always` with the specified policy (e.g., `IfNotPresent`). -- `SSH_GATEWAY_HOST` / `SSH_GATEWAY_PORT`: Replaces `__SSH_GATEWAY_HOST__` and `__SSH_GATEWAY_PORT__` placeholders. -- `EXTRA_SANS`: Builds a YAML flow-style list from the comma-separated SANs and replaces `extraSANs: []`. - -## Healthcheck Script - -`deploy/docker/cluster-healthcheck.sh` validates cluster readiness through a series of checks: - -1. **Kubernetes API**: `kubectl get --raw='/readyz'` -2. **OpenShell StatefulSet**: Checks that `statefulset/openshell` in namespace `openshell` exists and has 1 ready replica. -3. **Gateway**: Checks that `gateway/openshell-gateway` in namespace `openshell` has the `Programmed` condition. -4. **mTLS secret** (conditional): If `NAV_GATEWAY_TLS_ENABLED` is true (or inferred from the HelmChart manifest using the same two-path detection logic as the bootstrap code), checks that secret `openshell-cli-client` exists with non-empty `ca.crt`, `tls.crt`, and `tls.key` data. - -## GPU Enablement - -GPU support is part of the single-node gateway bootstrap path rather than a separate architecture. - -- `openshell gateway start --gpu` threads GPU device options through `crates/openshell-cli`, `crates/openshell-bootstrap`, and `crates/openshell-bootstrap/src/docker.rs`. -- When enabled, the cluster container is created with Docker `DeviceRequests`. The injection mechanism is selected based on whether CDI is enabled on the daemon (`SystemInfo.CDISpecDirs` via `GET /info`): - - **CDI enabled** (daemon reports non-empty `CDISpecDirs`): CDI device injection — `driver="cdi"` with `nvidia.com/gpu=all`. Specs are expected to be pre-generated on the host (e.g. automatically by the `nvidia-cdi-refresh.service` or manually via `nvidia-ctk generate`). - - **CDI not enabled**: `--gpus all` device request — `driver="nvidia"`, `count=-1`, which relies on the NVIDIA Container Runtime hook. -- `deploy/docker/Dockerfile.images` installs NVIDIA Container Toolkit packages in a dedicated Ubuntu stage and copies the runtime binaries, config, and `libnvidia-container` shared libraries into the final Ubuntu-based cluster image. -- `deploy/docker/cluster-entrypoint.sh` checks `GPU_ENABLED=true` and copies GPU-only manifests from `/opt/openshell/gpu-manifests/` into k3s's manifests directory. -- `deploy/kube/gpu-manifests/nvidia-device-plugin-helmchart.yaml` installs the NVIDIA device plugin chart, currently pinned to `0.18.2`. NFD and GFD are disabled; the device plugin's default `nodeAffinity` (which requires `feature.node.kubernetes.io/pci-10de.present=true` or `nvidia.com/gpu.present=true` from NFD/GFD) is overridden to empty so the DaemonSet schedules on the single-node cluster without requiring those labels. The chart is configured with `deviceListStrategy: cdi-cri` so the device plugin injects devices via direct CDI device requests in the CRI. -- k3s auto-detects `nvidia-container-runtime` on `PATH`, registers the `nvidia` containerd runtime, and creates the `nvidia` `RuntimeClass` automatically. -- The OpenShell Helm chart grants the gateway service account cluster-scoped read access to `node.k8s.io/runtimeclasses` and core `nodes` so GPU sandbox admission can verify both the `nvidia` `RuntimeClass` and allocatable GPU capacity before creating a sandbox. - -The runtime chain is: - -```text -Host GPU drivers & NVIDIA Container Toolkit - └─ Docker: DeviceRequests (CDI when enabled, --gpus all otherwise) - └─ k3s/containerd: nvidia-container-runtime on PATH -> auto-detected - └─ k8s: nvidia-device-plugin DaemonSet advertises nvidia.com/gpu - └─ Pods: request nvidia.com/gpu in resource limits (CDI injection — no runtimeClassName needed) -``` - -### `--gpu` flag - -The `--gpu` flag on `gateway start` enables GPU passthrough. OpenShell auto-selects CDI when enabled on the daemon and falls back to Docker's NVIDIA GPU request path (`--gpus all`) otherwise. +| Input | Purpose | +|---|---| +| Namespace | Release namespace and default sandbox namespace. | +| `openshell-ssh-handshake` Secret | HMAC key used by the SSH relay handshake. | +| `openshell-server-tls` Secret | Server certificate and key when TLS is enabled. | +| `openshell-server-client-ca` Secret | CA bundle used by the gateway to verify client certificates. | +| `openshell-client-tls` Secret | Client certificate bundle mounted into sandbox pods. | +| StorageClass / PVC support | Persistent gateway SQLite data when using the default `server.dbUrl`. | +| Service exposure | Port-forward, ingress, load balancer, or NodePort for CLI access. | -Device injection uses CDI (`deviceListStrategy: cdi-cri`): the device plugin injects devices via direct CDI device requests in the CRI. Sandbox pods only need `nvidia.com/gpu: 1` in their resource limits, and GPU pods do not set `runtimeClassName`. +For local Kubernetes evaluation, TLS may be disabled with `server.disableTls=true` and the service can be reached through `kubectl port-forward`. Production deployments should keep TLS enabled or place the gateway behind a trusted TLS-terminating access proxy. -The expected smoke test is a plain pod requesting `nvidia.com/gpu: 1` without `runtimeClassName` and running `nvidia-smi`. +Key Helm values: -## Remote Image Transfer +| Value | Effect | +|---|---| +| `image.repository`, `image.tag` | Select the gateway image. | +| `service.type`, `service.port`, `service.nodePort` | Expose the gateway service. | +| `server.dbUrl` | Select SQLite or Postgres persistence. | +| `server.sandboxNamespace` | Namespace for sandbox resources. | +| `server.sandboxImage` | Default sandbox image. | +| `server.grpcEndpoint` | Endpoint sandbox supervisors use to call back to the gateway. | +| `server.sshGatewayHost`, `server.sshGatewayPort` | Host and port returned to CLI clients for SSH proxy connections. | +| `server.disableTls`, `server.disableGatewayAuth` | Transport/authentication mode. | +| `server.tls.*` | Names of TLS secrets mounted into the gateway and sandboxes. | + +## Runtime Shape ```mermaid flowchart LR - B[bootstrap] -->|query| RP[Remote platform: Docker version API] - RP --> Auth[Authenticate with distribution registry] - Auth --> Pull[Pull platform-specific image layers] - Pull --> Tag[Tag to local image ref] - Tag --> OK[Image available as openshell/cluster:TAG] + CLI[openshell CLI] -->|gRPC / HTTP| GW[Gateway] + GW --> DB[(SQLite or Postgres)] + GW --> DRIVER[Compute Driver] + DRIVER --> DOCKER[Docker] + DRIVER --> PODMAN[Podman] + DRIVER --> K8S[Kubernetes API] + DRIVER --> VM[MicroVM Driver] + DOCKER --> SBX1[Sandbox Container] + PODMAN --> SBX2[Sandbox Container] + K8S --> SBX3[Sandbox Pod] + VM --> SBX4[Sandbox VM] + SBX1 --> GW + SBX2 --> GW + SBX3 --> GW + SBX4 --> GW ``` -- Remote platform is queried via `Docker::version()` and normalized (e.g., `x86_64` -> `amd64`, `aarch64` -> `arm64`). -- Distribution registry credentials are XOR-encoded in the binary (lightweight obfuscation, not a security boundary). -- If the image ref looks local (no `/` in repository), the `latest` tag is used from the distribution registry regardless of the local `IMAGE_TAG`. - -## Access Model - -### Gateway endpoint exposure - -- Local: `https://127.0.0.1:{port}` (or `https://{docker_host}:{port}` when `DOCKER_HOST` is a non-loopback TCP endpoint). Default port is 8080. -- Remote: `https://:{port}`. -- The host port (configurable via `--port`, default 8080) maps to container port 30051 (OpenShell service NodePort). - -## Lifecycle Operations - -### stop - -`GatewayHandle::stop()` calls `stop_container()`, which tolerates 404 (not found) and 409 (already stopped). - -### destroy - -**Bootstrap layer** (`GatewayHandle::destroy()` -> `destroy_gateway_resources()`): - -1. Stop the container. -2. Remove the container (`force=true`). Tolerates 404. -3. Remove the volume (`force=true`). Tolerates 404. -4. Force-remove the per-gateway network via `force_remove_network()`, disconnecting any stale endpoints first. - -**CLI layer** (`gateway_destroy()` in `run.rs` additionally): - -6. Remove the metadata JSON file via `remove_gateway_metadata()`. -7. Clear the active gateway reference if it matches the destroyed gateway. - -## Idempotency and Error Behavior - -- Re-running deploy is safe: - - Network is recreated on each deploy to guarantee a clean state; volume is reused (inspect before create). - - If a container exists with the same image ID, it is reused; if the image changed, the container is recreated. - - `start_container` tolerates already-running state (409). -- In interactive terminals, the CLI prompts the user to optionally destroy and recreate an existing gateway before redeploying. -- Error handling surfaces: - - Docker API failures from inspect/create/start/remove. - - SSH connection failures when creating the remote Docker client. - - Health check timeout (6 min) with recent container logs. - - Container exit during any polling phase (health, mTLS) with diagnostic information (exit code, OOM status, recent logs). - - mTLS secret polling timeout (3 min). - - Local image ref without registry prefix: clear error with build instructions rather than a failed Docker Hub pull. - -## Auto-Bootstrap from `sandbox create` - -When `openshell sandbox create` cannot connect to a gateway (connection refused, DNS error, missing default TLS certs), the CLI offers to bootstrap one automatically: - -1. `should_attempt_bootstrap()` in `crates/openshell-cli/src/bootstrap.rs` checks the error type. It returns `true` for connectivity errors and missing default TLS materials, but `false` for TLS handshake/auth errors. -2. If running in a terminal, the user is prompted to confirm. -3. `run_bootstrap()` deploys a gateway named `"openshell"`, sets it as active, and returns fresh `TlsOptions` pointing to the newly-written mTLS certs. -4. When `sandbox create` requests GPU explicitly (`--gpu`) or infers it from an image whose final name component contains `gpu` (such as `nvidia-gpu`), the bootstrap path enables gateway GPU support before retrying sandbox creation, using the same CDI-or-fallback selection as `gateway start --gpu`. - -## Container Environment Variables - -Variables set on the container by `ensure_container()` in `docker.rs`: - -| Variable | Value | When Set | -|---|---|---| -| `REGISTRY_MODE` | `"external"` | Always | -| `REGISTRY_HOST` | Distribution registry host (or `OPENSHELL_REGISTRY_HOST` override) | Always | -| `REGISTRY_INSECURE` | `"true"` or `"false"` | Always | -| `IMAGE_REPO_BASE` | `{registry_host}/{namespace}` (or `IMAGE_REPO_BASE`/`OPENSHELL_IMAGE_REPO_BASE` override) | Always | -| `REGISTRY_ENDPOINT` | Custom endpoint URL | When `OPENSHELL_REGISTRY_ENDPOINT` is set | -| `REGISTRY_USERNAME` | Registry auth username | When explicit credentials provided via `--registry-username`/`--registry-token` or env vars | -| `REGISTRY_PASSWORD` | Registry auth password | When explicit credentials provided via `--registry-username`/`--registry-token` or env vars | -| `EXTRA_SANS` | Comma-separated extra TLS SANs | When extra SANs computed | -| `SSH_GATEWAY_HOST` | Resolved remote hostname/IP | Remote deploys only | -| `SSH_GATEWAY_PORT` | Configured host port (default `8080`) | Remote deploys only | -| `IMAGE_TAG` | Image tag (e.g., `"dev"`) | When `IMAGE_TAG` env is set or push mode | -| `IMAGE_PULL_POLICY` | `"IfNotPresent"` | Push mode only | -| `PUSH_IMAGE_REFS` | Comma-separated image refs | Push mode only | - -## Host-Side Environment Variables - -Environment variables that affect bootstrap behavior when set on the host: - -| Variable | Effect | -|---|---| -| `OPENSHELL_CLUSTER_IMAGE` | Overrides entire image ref if set and non-empty | -| `IMAGE_TAG` | Sets image tag (default: `"dev"`) when `OPENSHELL_CLUSTER_IMAGE` is not set | -| `NAV_GATEWAY_TLS_ENABLED` | Overrides HelmChart manifest for TLS enabled check (`true`/`1`/`yes`/`false`/`0`/`no`) | -| `XDG_CONFIG_HOME` | Base config directory (default: `$HOME/.config`) | -| `DOCKER_HOST` | When `tcp://` and non-loopback, the host is added as a TLS SAN and used as the gateway endpoint | -| `OPENSHELL_PUSH_IMAGES` | Comma-separated image refs to push into the gateway's containerd (local deploy only) | -| `OPENSHELL_REGISTRY_HOST` | Override the distribution registry host | -| `OPENSHELL_REGISTRY_NAMESPACE` | Override the registry namespace (default: `"openshell"`) | -| `IMAGE_REPO_BASE` / `OPENSHELL_IMAGE_REPO_BASE` | Override the image repository base path | -| `OPENSHELL_REGISTRY_INSECURE` | Use HTTP instead of HTTPS for registry mirror | -| `OPENSHELL_REGISTRY_ENDPOINT` | Custom registry mirror endpoint | -| `OPENSHELL_REGISTRY_USERNAME` | Override registry auth username | -| `OPENSHELL_REGISTRY_PASSWORD` | Override registry auth password | -| `OPENSHELL_GATEWAY` | Set the active gateway name for CLI commands | - -## File System Layout - -Artifacts stored under `$XDG_CONFIG_HOME/openshell/` (default `~/.config/openshell/`): - -``` -openshell/ - active_gateway # plain text: active gateway name - gateways/ - {name}_metadata.json # GatewayMetadata JSON - {name}/ - mtls/ # mTLS bundle (when TLS enabled) - ca.crt - tls.crt - tls.key -``` +The gateway process manages all OpenShell control-plane APIs. It persists records in SQLite or Postgres, watches sandbox state through the selected compute driver, and brokers SSH access through supervisor-initiated relay streams. -## Implementation References +## Operational Notes -- `crates/openshell-bootstrap/src/lib.rs` -- public API, deploy orchestration -- `crates/openshell-bootstrap/src/docker.rs` -- Docker API wrappers -- `crates/openshell-bootstrap/src/image.rs` -- registry pull, XOR credentials -- `crates/openshell-bootstrap/src/runtime.rs` -- exec, health polling, stale node cleanup -- `crates/openshell-bootstrap/src/metadata.rs` -- metadata CRUD, active gateway, SSH resolution -- `crates/openshell-bootstrap/src/mtls.rs` -- TLS detection, secret extraction, atomic write -- `crates/openshell-bootstrap/src/push.rs` -- local image push into k3s containerd -- `crates/openshell-bootstrap/src/constants.rs` -- naming conventions -- `crates/openshell-bootstrap/src/paths.rs` -- XDG path helpers -- `crates/openshell-cli/src/main.rs` -- CLI command definitions -- `crates/openshell-cli/src/run.rs` -- CLI command implementations -- `crates/openshell-cli/src/bootstrap.rs` -- auto-bootstrap from sandbox create -- `deploy/docker/Dockerfile.images` -- shared image build definition (cluster target) -- `deploy/docker/cluster-entrypoint.sh` -- container entrypoint script -- `deploy/docker/cluster-healthcheck.sh` -- Docker HEALTHCHECK script -- `deploy/kube/manifests/openshell-helmchart.yaml` -- OpenShell Helm chart manifest -- `deploy/kube/manifests/envoy-gateway-helmchart.yaml` -- Envoy Gateway manifest +- Gateway endpoint registration should use `openshell gateway add ` regardless of compute platform. +- Kubernetes chart changes should be validated with `helm lint deploy/helm/openshell` and an install into a disposable namespace when possible. +- Docker driver changes should be validated with `mise run gateway:docker` or `mise run e2e:docker`. +- Podman driver changes should be validated with `mise run e2e:podman`. +- VM driver changes should be validated with `mise run e2e:vm`. +- Gateway image changes should be validated by building `deploy/docker/Dockerfile.images` target `gateway`. +- Published docs should describe gateway deployment and endpoint registration. diff --git a/architecture/gateway.md b/architecture/gateway.md index 9677dfc47e..c85f77ef61 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -2,7 +2,9 @@ ## Overview -`openshell-server` is the gateway -- the central control plane for a cluster. It exposes two gRPC services (OpenShell and Inference) and HTTP endpoints on a single multiplexed port, manages sandbox lifecycle through Kubernetes CRDs, persists state in SQLite or Postgres, and provides SSH tunneling into sandbox pods. The gateway coordinates all interactions between clients, the Kubernetes cluster, and the persistence layer. +`openshell-server` is the gateway -- the central control plane for a cluster. It exposes two gRPC services (OpenShell and Inference) and HTTP endpoints on a single multiplexed port, manages sandbox lifecycle through a pluggable compute driver, persists state in SQLite or Postgres, and brokers SSH access into sandboxes through supervisor-initiated relay streams. The gateway coordinates all interactions between clients, the compute backend, and the persistence layer. + +Each sandbox supervisor opens a persistent inbound gRPC session (`ConnectSupervisor`); the gateway multiplexes per-invocation `RelayStream` RPCs onto the same HTTP/2 connection to move bytes between clients and the in-sandbox SSH Unix socket. The gateway does not need to know, resolve, or reach the sandbox's network address. ## Architecture Diagram @@ -11,25 +13,27 @@ The following diagram shows the major components inside the gateway process and ```mermaid graph TD Client["gRPC / HTTP Client"] + Supervisor["Sandbox Supervisor
(inbound gRPC)"] TCP["TCP Listener"] TLS["TLS Acceptor
(optional)"] - MUX["MultiplexedService"] + MUX["MultiplexedService
(HTTP/2 adaptive window)"] GRPC_ROUTER["GrpcRouter"] NAV["OpenShellServer
(OpenShell service)"] INF["InferenceServer
(Inference service)"] HTTP["HTTP Router
(Axum)"] HEALTH["Health Endpoints"] SSH_TUNNEL["SSH Tunnel
(/connect/ssh)"] + SUP_REG["SupervisorSessionRegistry"] STORE["Store
(SQLite / Postgres)"] - K8S["Kubernetes API"] - WATCHER["Sandbox Watcher"] - EVENT_TAILER["Kube Event Tailer"] + COMPUTE["ComputeRuntime"] + DRIVER["ComputeDriver
(kubernetes / vm)"] WATCH_BUS["SandboxWatchBus"] LOG_BUS["TracingLogBus"] PLAT_BUS["PlatformEventBus"] INDEX["SandboxIndex"] Client --> TCP + Supervisor --> TCP TCP --> TLS TLS --> MUX MUX -->|"content-type: application/grpc"| GRPC_ROUTER @@ -39,17 +43,16 @@ graph TD HTTP --> HEALTH HTTP --> SSH_TUNNEL NAV --> STORE - NAV --> K8S - INF --> STORE + NAV --> COMPUTE + NAV --> SUP_REG SSH_TUNNEL --> STORE - SSH_TUNNEL --> K8S - WATCHER --> K8S - WATCHER --> STORE - WATCHER --> WATCH_BUS - WATCHER --> INDEX - EVENT_TAILER --> K8S - EVENT_TAILER --> PLAT_BUS - EVENT_TAILER --> INDEX + SSH_TUNNEL --> SUP_REG + INF --> STORE + COMPUTE --> DRIVER + COMPUTE --> STORE + COMPUTE --> WATCH_BUS + COMPUTE --> INDEX + COMPUTE --> PLAT_BUS LOG_BUS --> PLAT_BUS ``` @@ -57,21 +60,24 @@ graph TD | Module | File | Purpose | |--------|------|---------| -| Entry point | `crates/openshell-server/src/main.rs` | CLI argument parsing, config assembly, tracing setup, calls `run_server` | +| Entry point | `crates/openshell-server/src/main.rs` | Thin binary wrapper that calls `cli::run_cli` | +| CLI | `crates/openshell-server/src/cli.rs` | `Args` parser, config assembly, tracing setup, calls `run_server` | | Gateway runtime | `crates/openshell-server/src/lib.rs` | `ServerState` struct, `run_server()` accept loop | -| Protocol mux | `crates/openshell-server/src/multiplex.rs` | `MultiplexService`, `MultiplexedService`, `GrpcRouter`, `BoxBody` | -| gRPC: OpenShell | `crates/openshell-server/src/grpc.rs` | `OpenShellService` -- sandbox CRUD, provider CRUD, watch, exec, SSH sessions, policy delivery | -| gRPC: Inference | `crates/openshell-server/src/inference.rs` | `InferenceService` -- cluster inference config (set/get) and sandbox inference bundle delivery | +| Protocol mux | `crates/openshell-server/src/multiplex.rs` | `MultiplexService`, `MultiplexedService`, `GrpcRouter`, `BoxBody`, HTTP/2 adaptive-window tuning | +| gRPC: OpenShell | `crates/openshell-server/src/grpc/mod.rs` | `OpenShellService` trait impl -- dispatches to per-concern handlers | +| gRPC: Sandbox/Exec | `crates/openshell-server/src/grpc/sandbox.rs` | Sandbox CRUD, `ExecSandbox`, SSH session handlers, relay-backed exec proxy | +| gRPC: Inference | `crates/openshell-server/src/inference.rs` | `InferenceService` -- gateway inference config and sandbox bundle delivery | +| Supervisor sessions | `crates/openshell-server/src/supervisor_session.rs` | `SupervisorSessionRegistry`, `handle_connect_supervisor`, `handle_relay_stream`, reaper | | HTTP | `crates/openshell-server/src/http.rs` | Health endpoints, merged with SSH tunnel router | | Browser auth | `crates/openshell-server/src/auth.rs` | Cloudflare browser login relay at `/auth/connect` | -| SSH tunnel | `crates/openshell-server/src/ssh_tunnel.rs` | HTTP CONNECT handler at `/connect/ssh` | +| SSH tunnel | `crates/openshell-server/src/ssh_tunnel.rs` | HTTP CONNECT handler at `/connect/ssh` backed by `open_relay` | | WS tunnel | `crates/openshell-server/src/ws_tunnel.rs` | WebSocket tunnel handler at `/_ws_tunnel` for Cloudflare-fronted clients | | TLS | `crates/openshell-server/src/tls.rs` | `TlsAcceptor` wrapping rustls with ALPN | | Persistence | `crates/openshell-server/src/persistence/mod.rs` | `Store` enum (SQLite/Postgres), generic object CRUD, protobuf codec | -| Persistence: SQLite | `crates/openshell-server/src/persistence/sqlite.rs` | `SqliteStore` with sqlx | -| Persistence: Postgres | `crates/openshell-server/src/persistence/postgres.rs` | `PostgresStore` with sqlx | | Compute runtime | `crates/openshell-server/src/compute/mod.rs` | `ComputeRuntime`, gateway-owned sandbox lifecycle orchestration over a compute backend | -| Compute driver: Kubernetes | `crates/openshell-driver-kubernetes/src/driver.rs` | Kubernetes CRD create/delete, endpoint resolution, watch stream, pod template translation | +| Compute driver: Kubernetes | `crates/openshell-driver-kubernetes/src/driver.rs` | Kubernetes CRD create/delete/watch, pod template translation | +| Compute driver: Docker | `crates/openshell-driver-docker/src/lib.rs` | Local Docker container create/stop/delete/watch | +| Compute driver: VM | `crates/openshell-driver-vm/src/driver.rs` | Per-sandbox microVM create/delete/watch, supervisor-only guest boot | | Sandbox index | `crates/openshell-server/src/sandbox_index.rs` | `SandboxIndex` -- in-memory name/pod-to-id correlation | | Watch bus | `crates/openshell-server/src/sandbox_watch.rs` | `SandboxWatchBus` -- in-memory broadcast for persisted sandbox updates | | Tracing bus | `crates/openshell-server/src/tracing_bus.rs` | `TracingLogBus` -- captures tracing events keyed by `sandbox_id` | @@ -80,38 +86,46 @@ Proto definitions consumed by the gateway: | Proto file | Package | Defines | |------------|---------|---------| -| `proto/openshell.proto` | `openshell.v1` | `OpenShell` service, public sandbox resource model, provider/SSH/watch messages | -| `proto/compute_driver.proto` | `openshell.compute.v1` | Internal `ComputeDriver` service, driver-native sandbox observations, endpoint resolution, compute watch stream envelopes | +| `proto/openshell.proto` | `openshell.v1` | `OpenShell` service, public sandbox resource model, provider/SSH/watch/policy messages, supervisor session messages (`ConnectSupervisor`, `RelayStream`, `RelayFrame`) | +| `proto/compute_driver.proto` | `openshell.compute.v1` | Internal `ComputeDriver` service, driver-native sandbox observations, compute watch stream envelopes | | `proto/inference.proto` | `openshell.inference.v1` | `Inference` service: `SetClusterInference`, `GetClusterInference`, `GetInferenceBundle` | | `proto/datamodel.proto` | `openshell.datamodel.v1` | `Provider` | | `proto/sandbox.proto` | `openshell.sandbox.v1` | Sandbox supervisor policy, settings, and config messages | ## Startup Sequence -The gateway boots in `main()` (`crates/openshell-server/src/main.rs`) and proceeds through these steps: +The gateway boots in `cli::run_cli` (`crates/openshell-server/src/cli.rs`) and proceeds through these steps: -1. **Install rustls crypto provider** -- `aws_lc_rs::default_provider().install_default()`. +1. **Install rustls crypto provider** -- `rustls::crypto::ring::default_provider().install_default()`. 2. **Parse CLI arguments** -- `Args::parse()` via `clap`. Every flag has a corresponding environment variable (see [Configuration](#configuration)). 3. **Initialize tracing** -- Creates a `TracingLogBus` and installs a tracing subscriber that writes to stdout and publishes log events keyed by `sandbox_id` into the bus. -4. **Build `Config`** -- Assembles a `openshell_core::Config` from the parsed arguments. +4. **Build `Config`** -- Assembles an `openshell_core::Config` from the parsed arguments. 5. **Call `run_server()`** (`crates/openshell-server/src/lib.rs`): 1. Connect to the persistence store (`Store::connect`), which auto-detects SQLite vs Postgres from the URL prefix and runs migrations. - 2. Create `ComputeRuntime` with an in-process `ComputeDriverService` backed by `KubernetesComputeDriver`, so the gateway calls the `openshell.compute.v1.ComputeDriver` RPC surface even without transport. - 3. Build `ServerState` (shared via `Arc` across all handlers). - 4. **Spawn background tasks**: - - `ComputeRuntime::spawn_watchers` -- consumes the compute-driver watch stream, republishes platform events, and runs a periodic `ListSandboxes` snapshot reconcile so the store-backed public sandbox reads stay aligned with the compute driver. - 5. Create `MultiplexService`. - 6. Bind `TcpListener` on `config.bind_address`. - 7. Optionally create `TlsAcceptor` from cert/key files. - 8. Enter the accept loop: for each connection, spawn a tokio task that optionally performs a TLS handshake, then calls `MultiplexService::serve()`. + 2. Create `ComputeRuntime` with a `ComputeDriver` implementation selected by `OPENSHELL_DRIVERS`: + - `kubernetes` wraps `KubernetesComputeDriver` in `ComputeDriverService`, so the gateway uses the `openshell.compute.v1.ComputeDriver` RPC surface even without transport. + - `docker` constructs `openshell-driver-docker` in-process and manages local containers labeled with the configured sandbox namespace. + - `vm` spawns the standalone `openshell-driver-vm` binary as a local compute-driver process, resolves it from `--driver-dir`, conventional libexec install paths, or a sibling of the gateway binary, connects to it over a Unix domain socket, and keeps the libkrun/rootfs runtime out of the gateway binary. + 3. Build `ServerState` (shared via `Arc` across all handlers), including a fresh `SupervisorSessionRegistry`. + 4. Resume persisted sandboxes that were stopped during the previous gateway shutdown. + 5. **Spawn background tasks**: + - `ComputeRuntime::spawn_watchers` -- consumes the compute-driver watch stream, republishes platform events, and runs a periodic `ListSandboxes` snapshot reconcile. + - `ssh_tunnel::spawn_session_reaper` -- sweeps expired or revoked SSH session tokens from the store hourly. + - `supervisor_session::spawn_relay_reaper` -- sweeps orphaned pending relay channels every 30 seconds. + 6. Create `MultiplexService`. + 7. Bind the primary gateway listener and any compute-driver requested listeners. Docker requests the Docker bridge gateway address with the normal gateway port, so sandbox containers can call back over the bridge without joining the host network. + 8. Bind optional health and metrics listeners. + 9. Optionally create `TlsAcceptor` from cert/key files. + 10. Spawn a task per gateway listener. Each accepted connection optionally performs a TLS handshake, then calls `MultiplexService::serve()`. ## Configuration -All configuration is via CLI flags with environment variable fallbacks. The `--db-url` flag is the only required argument. +All configuration is via CLI flags with environment variable fallbacks. The `--db-url` flag is required. | Flag | Env Var | Default | Description | |------|---------|---------|-------------| -| `--port` | `OPENSHELL_SERVER_PORT` | `8080` | TCP listen port (binds `0.0.0.0`) | +| `--bind-address` | `OPENSHELL_BIND_ADDRESS` | `127.0.0.1` | IP address for gateway, health, and metrics listeners. Container deployments pass `0.0.0.0` explicitly. | +| `--port` | `OPENSHELL_SERVER_PORT` | `8080` | TCP listen port | | `--log-level` | `OPENSHELL_LOG_LEVEL` | `info` | Tracing log level filter | | `--tls-cert` | `OPENSHELL_TLS_CERT` | None | Path to PEM certificate file | | `--tls-key` | `OPENSHELL_TLS_KEY` | None | Path to PEM private key file | @@ -122,13 +136,22 @@ All configuration is via CLI flags with environment variable fallbacks. The `--d | `--db-url` | `OPENSHELL_DB_URL` | *required* | Database URL (`sqlite:...` or `postgres://...`). The Helm chart defaults to `sqlite:/var/openshell/openshell.db` (persistent volume). In-memory SQLite (`sqlite::memory:?cache=shared`) works for ephemeral/test environments but data is lost on restart. | | `--sandbox-namespace` | `OPENSHELL_SANDBOX_NAMESPACE` | `default` | Kubernetes namespace for sandbox CRDs | | `--sandbox-image` | `OPENSHELL_SANDBOX_IMAGE` | None | Default container image for sandbox pods | -| `--grpc-endpoint` | `OPENSHELL_GRPC_ENDPOINT` | None | gRPC endpoint reachable from within the cluster (for sandbox callbacks) | +| `--grpc-endpoint` | `OPENSHELL_GRPC_ENDPOINT` | None | gRPC endpoint reachable from sandbox workloads for supervisor callbacks | +| `--drivers` | `OPENSHELL_DRIVERS` | `kubernetes` | Compute backend to use. Current options are `kubernetes`, `docker`, and `vm`. | +| `--docker-network-name` | `OPENSHELL_DOCKER_NETWORK_NAME` | `openshell-docker` | Docker bridge network that local Docker sandboxes join | +| `--driver-dir` | `OPENSHELL_DRIVER_DIR` | unset | Override directory for `openshell-driver-vm`. When unset, the gateway searches `~/.local/libexec/openshell`, `/usr/libexec/openshell`, `/usr/local/libexec/openshell`, `/usr/local/libexec`, then a sibling binary. | +| `--vm-driver-state-dir` | `OPENSHELL_VM_DRIVER_STATE_DIR` | `target/openshell-vm-driver` | Host directory for VM sandbox rootfs, console logs, runtime state, and shared image-rootfs cache | +| `--vm-krun-log-level` | `OPENSHELL_VM_KRUN_LOG_LEVEL` | `1` | libkrun log level for VM helper processes | +| `--vm-driver-vcpus` | `OPENSHELL_VM_DRIVER_VCPUS` | `2` | Default vCPU count for VM sandboxes | +| `--vm-driver-mem-mib` | `OPENSHELL_VM_DRIVER_MEM_MIB` | `2048` | Default memory allocation for VM sandboxes in MiB | +| `--vm-tls-ca` | `OPENSHELL_VM_TLS_CA` | None | CA cert copied into VM guests for gateway mTLS | +| `--vm-tls-cert` | `OPENSHELL_VM_TLS_CERT` | None | Client cert copied into VM guests for gateway mTLS | +| `--vm-tls-key` | `OPENSHELL_VM_TLS_KEY` | None | Client private key copied into VM guests for gateway mTLS | | `--ssh-gateway-host` | `OPENSHELL_SSH_GATEWAY_HOST` | `127.0.0.1` | Public hostname returned in SSH session responses | | `--ssh-gateway-port` | `OPENSHELL_SSH_GATEWAY_PORT` | `8080` | Public port returned in SSH session responses | | `--ssh-connect-path` | `OPENSHELL_SSH_CONNECT_PATH` | `/connect/ssh` | HTTP path for SSH CONNECT/upgrade | -| `--sandbox-ssh-port` | `OPENSHELL_SANDBOX_SSH_PORT` | `2222` | SSH listen port inside sandbox pods | -| `--ssh-handshake-secret` | `OPENSHELL_SSH_HANDSHAKE_SECRET` | None | Shared HMAC-SHA256 secret for gateway-to-sandbox handshake | -| `--ssh-handshake-skew-secs` | `OPENSHELL_SSH_HANDSHAKE_SKEW_SECS` | `300` | Allowed clock skew (seconds) for SSH handshake timestamps | + +The sandbox-side SSH listener is a Unix domain socket inside the sandbox. The path defaults to `/run/openshell/ssh.sock` and is configured on the compute driver (e.g. `openshell-driver-kubernetes --sandbox-ssh-socket-path`). The gateway never dials this socket itself; the supervisor bridges it onto a `RelayStream` when asked. ## Shared State @@ -145,15 +168,17 @@ pub struct ServerState { pub ssh_connections_by_token: Mutex>, pub ssh_connections_by_sandbox: Mutex>, pub settings_mutex: tokio::sync::Mutex<()>, + pub supervisor_sessions: SupervisorSessionRegistry, } ``` - **`store`** -- persistence backend (SQLite or Postgres) for all object types. -- **`compute`** -- gateway-owned compute orchestration. Persists sandbox lifecycle transitions, validates create requests through the compute backend, resolves exec/SSH endpoints, consumes the backend watch stream, and periodically reconciles the store against `ComputeDriver/ListSandboxes` snapshots. +- **`compute`** -- gateway-owned compute orchestration. Persists sandbox lifecycle transitions, validates create requests through the compute backend, consumes the backend watch stream, and periodically reconciles the store against `ComputeDriver/ListSandboxes` snapshots. - **`sandbox_index`** -- in-memory bidirectional index mapping sandbox names and agent pod names to sandbox IDs. Updated from compute-driver sandbox snapshots. - **`sandbox_watch_bus`** -- `broadcast`-based notification bus keyed by sandbox ID. Producers call `notify(&id)` when the persisted sandbox record changes; consumers in `WatchSandbox` streams receive `()` signals and re-read the record. - **`tracing_log_bus`** -- captures `tracing` events that include a `sandbox_id` field and republishes them as `SandboxLogLine` messages. Maintains a per-sandbox tail buffer (default 200 entries). Also contains a nested `PlatformEventBus` for compute-driver platform events. -- **`settings_mutex`** -- serializes settings mutations (global and sandbox) to prevent read-modify-write races. Held for the duration of any setting set/delete or global policy set/delete operation. See [Gateway Settings Channel](gateway-settings.md#global-policy-lifecycle). +- **`supervisor_sessions`** -- tracks the live `ConnectSupervisor` session per sandbox and the set of pending relay channels awaiting the supervisor's `RelayStream` dial-back. See [Supervisor Sessions](#supervisor-sessions). +- **`settings_mutex`** -- serializes settings mutations (global and sandbox) to prevent read-modify-write races. See [Gateway Settings Channel](gateway-settings.md#global-policy-lifecycle). ## Protocol Multiplexing @@ -164,8 +189,9 @@ All traffic (gRPC and HTTP) shares a single TCP port. Multiplexing happens at th `MultiplexService::serve()` (`crates/openshell-server/src/multiplex.rs`) creates per-connection service instances: 1. Each accepted TCP stream (optionally TLS-wrapped) is passed to `hyper_util::server::conn::auto::Builder`, which auto-negotiates HTTP/1.1 or HTTP/2. -2. The builder calls `serve_connection_with_upgrades()`, which supports HTTP upgrades (needed for the SSH tunnel's CONNECT method). -3. For each request, `MultiplexedService` inspects the `content-type` header: +2. The HTTP/2 side is built with `adaptive_window(true)`. Hyper/h2 auto-sizes the per-stream flow-control window based on measured bandwidth-delay product, so bulk byte transfers on `RelayStream` (and `ExecSandbox` / `PushSandboxLogs`) are not throttled by the default 64 KiB window. Idle streams stay cheap; active streams grow as needed. +3. The builder calls `serve_connection_with_upgrades()`, which supports HTTP upgrades (needed for the SSH tunnel's CONNECT method). +4. For each request, `MultiplexedService` inspects the `content-type` header: - **Starts with `application/grpc`** -- routes to `GrpcRouter`. - **Anything else** -- routes to the Axum HTTP router. @@ -189,33 +215,148 @@ When TLS is enabled (`crates/openshell-server/src/tls.rs`): - `--disable-tls` removes gateway-side TLS entirely and serves plaintext HTTP behind a trusted reverse proxy or tunnel. - Supports PKCS#1, PKCS#8, and SEC1 private key formats. - The TLS handshake happens before the stream reaches Hyper's auto builder, so ALPN negotiation and HTTP version detection work together transparently. -- Certificates are generated at cluster bootstrap time by the `openshell-bootstrap` crate using `rcgen`, not by a Helm Job. The bootstrap reconciles three K8s secrets: `openshell-server-tls` (server cert+key), `openshell-server-client-ca` (CA cert), and `openshell-client-tls` (client cert+key+CA, shared by CLI and sandbox pods). -- **Certificate lifetime**: Certificates use `rcgen` defaults (effectively never expire), which is appropriate for an internal dev-cluster PKI where certs are ephemeral to the cluster's lifetime. -- **Redeploy behavior**: On redeploy, existing cluster TLS secrets are loaded and reused if they are complete and valid PEM. If secrets are missing, incomplete, or malformed, fresh PKI is generated. If rotation occurs and the openshell workload is already running, the bootstrap performs a rollout restart and waits for completion before persisting CLI-side credentials. +- Certificates are operator-provided in the target deployment model. Helm deployments consume three K8s secrets: `openshell-server-tls` (server cert+key), `openshell-server-client-ca` (CA cert), and `openshell-client-tls` (client cert+key+CA, shared by CLI and sandbox workloads). +- Sandbox supervisors reuse the shared client cert to authenticate their `ConnectSupervisor` and `RelayStream` calls over the same mTLS channel. + +## Supervisor Sessions + +The gateway brokers all byte-level access into a sandbox through a two-plane design on a single HTTP/2 connection initiated by the supervisor: + +1. **Control plane** -- `ConnectSupervisor(stream SupervisorMessage) returns (stream GatewayMessage)`. Long-lived, one per sandbox. Carries `SupervisorHello`, `SessionAccepted`/`SessionRejected`, heartbeats, and `RelayOpen`/`RelayClose` control messages. +2. **Data plane** -- `RelayStream(stream RelayFrame) returns (stream RelayFrame)`. One short-lived call per SSH or exec invocation. The first inbound frame is a `RelayInit { channel_id }`; subsequent frames carry raw bytes in `RelayFrame.data` in either direction. + +Both RPCs are defined in `proto/openshell.proto` and ride the same TCP + TLS + HTTP/2 connection from the supervisor. No new TLS handshake, no reverse HTTP CONNECT, no direct gateway-to-pod dial. + +### `SupervisorSessionRegistry` + +`crates/openshell-server/src/supervisor_session.rs` defines `SupervisorSessionRegistry`, a single instance of which lives on `ServerState.supervisor_sessions`. It holds two maps guarded by `std::sync::Mutex`: + +- `sessions: HashMap` -- one entry per connected supervisor. Each `LiveSession` carries a unique `session_id`, the `mpsc::Sender` for the outbound stream, and a connection timestamp. +- `pending_relays: HashMap` -- one entry per in-flight `open_relay` call awaiting the supervisor's `RelayStream` dial-back. Each `PendingRelay` wraps a `oneshot::Sender` and a creation timestamp. + +Core operations: + +| Method | Purpose | +|--------|---------| +| `register(sandbox_id, session_id, tx)` | Insert a live session; returns the previous session's sender (if any) so the caller can close it. Used by `handle_connect_supervisor` when a supervisor reconnects. | +| `remove_if_current(sandbox_id, session_id)` | Remove the session only if its `session_id` still matches. Guards against the supersede race where an old session's cleanup task fires after a newer session already registered. | +| `open_relay(sandbox_id, session_wait_timeout)` | Wait up to `session_wait_timeout` for a live session, allocate a fresh `channel_id` (UUID v4), insert the pending slot, send `RelayOpen { channel_id }` to the supervisor, and return `(channel_id, oneshot::Receiver)`. The receiver resolves once the supervisor's `RelayStream` arrives and `claim_relay` pairs them up. | +| `claim_relay(channel_id)` | Consume the pending slot, construct a `tokio::io::duplex(64 KiB)` pair, hand the gateway-side half to the waiter via the oneshot, and return the supervisor-side half to `handle_relay_stream`. | +| `reap_expired_relays()` | Drop pending relays older than 10 s. Called by `spawn_relay_reaper` on a 30 s cadence. | + +Session wait uses exponential backoff from 100 ms to 2 s while polling the sessions map. Pending-relay expiry is fixed at `RELAY_PENDING_TIMEOUT = 10 s`. + +### `handle_connect_supervisor` + +Lifecycle of a supervisor session: + +1. Read the first `SupervisorMessage`; require `payload = Hello { sandbox_id, instance_id }` and a non-empty `sandbox_id`. +2. Allocate a fresh `session_id` (UUID v4) and create an `mpsc::channel::(64)` for the outbound stream. +3. Call `registry.register(...)`. If it returns a previous sender, log that the previous session was superseded (dropping the previous `tx` closes the old outbound stream). +4. Send `SessionAccepted { session_id, heartbeat_interval_secs: 15 }`. If the send fails, call `remove_if_current` (so a concurrent reconnect isn't evicted) and return `Internal`. +5. Spawn a session loop that `select!`s between inbound messages and a 15 s heartbeat timer. Inbound heartbeats are silent; `RelayOpenResult` is logged; `RelayClose` is logged; unknown payloads are logged as warnings. +6. When the loop exits (inbound EOF, inbound error, or outbound channel closed), `remove_if_current` drops the registration -- unless a newer session has already replaced it. + +### `handle_relay_stream` + +Lifecycle of one relay call: + +1. Read the first inbound `RelayFrame`; require `payload = Init { channel_id }` with a non-empty `channel_id`. Reject anything else with `InvalidArgument`. +2. Call `registry.claim_relay(channel_id)`. Returns `NotFound` if the channel is unknown or already expired, `DeadlineExceeded` if older than 10 s, or `Internal` if the waiter has dropped the oneshot receiver. +3. Split the supervisor-side `DuplexStream` into read and write halves and spawn two tasks: + - **Supervisor → gateway**: pull `RelayFrame`s from the inbound stream, accept `Data(bytes)`, write to the duplex write-half. On non-data frames, warn and break. Best-effort `shutdown()` on exit so the reader sees EOF. + - **Gateway → supervisor**: read up to `RELAY_STREAM_CHUNK_SIZE = 16 KiB` at a time from the duplex read-half and emit `RelayFrame { Data }` messages on an outbound `mpsc::channel(16)`. +4. Return the outbound receiver as the RPC response stream. + +### Connect Flow (SSH Tunnel) + +```mermaid +sequenceDiagram + participant Client as SSH client + participant GW as Gateway
(/connect/ssh) + participant Reg as SupervisorSessionRegistry + participant Sup as Sandbox Supervisor + participant Daemon as In-sandbox sshd
(Unix socket) + + Client->>GW: CONNECT /connect/ssh
x-sandbox-id, x-sandbox-token + GW->>GW: validate session + sandbox Ready + GW->>Reg: open_relay(sandbox_id, 30s) + Reg->>Sup: GatewayMessage::RelayOpen { channel_id } + Note over Reg: waits for RelayStream on channel_id + Sup->>Daemon: connect to Unix socket + Sup->>GW: RelayStream(RelayFrame::Init { channel_id }) + GW->>Reg: claim_relay(channel_id) + Reg-->>Sup: supervisor-side DuplexStream + Reg-->>GW: gateway-side DuplexStream + GW-->>Client: 200 OK + HTTP upgrade + Client<<->>GW: copy_bidirectional(upgraded, duplex) + GW<<->>Sup: RelayFrame::Data in both directions + Sup<<->>Daemon: raw SSH bytes +``` + +Timeouts on the tunnel path: + +- `open_relay` session wait: **30 s**. A first `sandbox connect` immediately after `sandbox create` must cover the supervisor's initial TLS + gRPC handshake on a cold pod. +- `relay_rx` delivery timeout: 10 s. Covers the round-trip from the `RelayOpen` message to the supervisor's `RelayStream` dial-back. + +Per-token and per-sandbox concurrent-tunnel limits (3 and 20 respectively) are still enforced before the upgrade. + +### Exec Flow + +`ExecSandbox` reuses the same machinery from `grpc/sandbox.rs`: + +1. Validate the request (`sandbox_id`, `command`, env-key format, other field rules), fetch the sandbox, require `Ready` phase. +2. `state.supervisor_sessions.open_relay(&sandbox.id, 15s)` -- shorter timeout than SSH connect, because exec is typically called mid-lifetime after the supervisor session is already established. +3. Wait up to 10 s for the relay `DuplexStream`. +4. `stream_exec_over_relay`: bind an ephemeral localhost TCP listener, bridge that single-use TCP socket to the relay duplex, and drive a `russh` client through the local port. The `russh` session opens a channel, executes the shell-escaped command, and streams `ExecSandboxStdout`/`ExecSandboxStderr` chunks to the caller. On completion, send `ExecSandboxExit { exit_code }`. +5. On timeout (if `timeout_seconds > 0`), emit exit code 124 (matching `timeout(1)`). + +The supervisor-side SSH daemon is an SSH server bound to a Unix domain socket inside the sandbox's filesystem. Filesystem permissions on that socket are the only access-control boundary between the supervisor bridge and the daemon; all higher-level authorization is enforced at `CreateSshSession` / `ExecSandbox` in the gateway. + +### Regression Coverage + +`crates/openshell-server/tests/supervisor_relay_integration.rs` is the regression guard for the `RelayStream` wire protocol. It stands up an in-process tonic server that mounts the real `handle_relay_stream` behind `MultiplexedService`, connects a mock supervisor client over a real tonic `Channel`, and exercises the registry's `open_relay` → `claim_relay` pairing end to end with `tokio::io::duplex` bridging. The five test cases cover: + +- Round-trip bytes from gateway to supervisor and back (echo loop). +- Clean close when the gateway drops the relay. +- EOF propagation when the supervisor closes its outbound sender. +- `Unavailable` when `open_relay` is called without a registered session. +- Concurrent `RelayStream` calls multiplexed independently on the same connection. + +These complement the unit tests inside `supervisor_session.rs` (registry-only behavior) and the live cluster tests (full CLI → gateway → sandbox path). ## gRPC Services ### OpenShell Service -Defined in `proto/openshell.proto`, implemented in `crates/openshell-server/src/grpc.rs` as `OpenShellService`. +Defined in `proto/openshell.proto`, implemented in `crates/openshell-server/src/grpc/mod.rs` as `OpenShellService`. Per-concern handlers live in `crates/openshell-server/src/grpc/` submodules. #### Sandbox Management | RPC | Description | Key behavior | |-----|-------------|--------------| | `Health` | Returns service status and version | Always returns `HEALTHY` with `CARGO_PKG_VERSION` | -| `CreateSandbox` | Create a new sandbox | Validates spec and policy, validates provider names exist (fail-fast), persists to store, creates Kubernetes CRD. On K8s 409 conflict or error, rolls back the store record and index entry. | +| `CreateSandbox` | Create a new sandbox | Validates spec and policy, validates provider names exist (fail-fast), persists to store, creates the compute-driver sandbox. On driver failure, rolls back the store record and index entry. | | `GetSandbox` | Fetch sandbox by name | Looks up by name via `store.get_message_by_name()` | | `ListSandboxes` | List sandboxes | Paginated (default limit 100), decodes protobuf payloads from store records | -| `DeleteSandbox` | Delete sandbox by name | Sets phase to `Deleting`, persists, notifies watch bus, then deletes the Kubernetes CRD. Cleans up store if the CRD was already gone. | +| `DeleteSandbox` | Delete sandbox by name | Sets phase to `Deleting`, persists, notifies watch bus, then deletes via the compute driver. Cleans up store if the sandbox was already gone. | | `WatchSandbox` | Stream sandbox updates | Server-streaming RPC. See [Watch Sandbox Stream](#watch-sandbox-stream) below. | -| `ExecSandbox` | Execute command in sandbox | Server-streaming RPC. See [Remote Exec via SSH](#remote-exec-via-ssh) below. | +| `ExecSandbox` | Execute command in sandbox | Server-streaming RPC; data plane runs through `SupervisorSessionRegistry::open_relay`. See [Exec Flow](#exec-flow). | + +#### Supervisor Session + +| RPC | Description | +|-----|-------------| +| `ConnectSupervisor` | Persistent bidi stream from the sandbox supervisor. Carries hello/accept/heartbeat/`RelayOpen`/`RelayClose`. One session per sandbox; reconnects supersede. | +| `RelayStream` | Per-invocation bidi byte bridge. Supervisor initiates after receiving `RelayOpen`; first frame is `RelayInit { channel_id }`; subsequent frames carry raw bytes. | + +Neither RPC is called by end users. They are the private control/data plane between the gateway and each sandbox supervisor. #### SSH Session Management | RPC | Description | |-----|-------------| -| `CreateSshSession` | Creates a session token for a `Ready` sandbox. Persists an `SshSession` record and returns gateway connection details (host, port, scheme, connect path). | +| `CreateSshSession` | Creates a session token for a `Ready` sandbox. Persists an `SshSession` record and returns gateway connection details (host, port, scheme, connect path). The resulting token is presented on the `/connect/ssh` HTTP CONNECT request. | | `RevokeSshSession` | Marks a session as revoked by setting `session.revoked = true` in the store. | #### Provider Management @@ -232,17 +373,17 @@ Full CRUD for `Provider` objects, which store typed credentials (e.g., API keys #### Policy, Settings, and Provider Environment Delivery -These RPCs are called by sandbox pods at startup and during runtime polling. +These RPCs are called by sandbox supervisors at startup and during runtime polling. | RPC | Description | |-----|-------------| -| `GetSandboxSettings` | Returns effective sandbox config looked up by sandbox ID: policy payload, policy metadata (version, hash, source, `global_policy_version`), merged effective settings, and a `config_revision` fingerprint for change detection. Two-tier resolution: registered keys start unset, sandbox values overlay, global values override. The reserved `policy` key in global settings can override the sandbox's own policy. When a global policy is active, `policy_source` is `GLOBAL` and `global_policy_version` carries the active revision number. See [Gateway Settings Channel](gateway-settings.md). | -| `GetGatewaySettings` | Returns gateway-global settings only (excluding the reserved `policy` key). Returns registered keys with empty values when unconfigured, and a monotonic `settings_revision`. | +| `GetSandboxConfig` | Returns effective sandbox config looked up by sandbox ID: policy payload, policy metadata (version, hash, source, `global_policy_version`), merged effective settings, and a `config_revision` fingerprint for change detection. Two-tier resolution: registered keys start unset, sandbox values overlay, global values override. The reserved `policy` key in global settings can override the sandbox's own policy. When a global policy is active, `policy_source` is `GLOBAL` and `global_policy_version` carries the active revision number. See [Gateway Settings Channel](gateway-settings.md). | +| `GetGatewayConfig` | Returns gateway-global settings only (excluding the reserved `policy` key). Returns registered keys with empty values when unconfigured, and a monotonic `settings_revision`. | | `GetSandboxProviderEnvironment` | Resolves provider credentials into environment variables for a sandbox. Iterates the sandbox's `spec.providers` list, fetches each `Provider`, and collects credential key-value pairs. First provider wins on duplicate keys. Skips credential keys that do not match `^[A-Za-z_][A-Za-z0-9_]*$`. | #### Policy Recommendation (Network Rules) -These RPCs support the sandbox-initiated policy recommendation pipeline. The sandbox generates proposals via its mechanistic mapper and submits them; the gateway validates, persists, and manages the approval workflow. See [architecture/policy-advisor.md](policy-advisor.md) for the full pipeline design. +These RPCs support the sandbox-initiated policy recommendation pipeline. The sandbox generates proposals via its mechanistic mapper and submits them; the gateway validates, persists, and manages the approval workflow. See [policy-advisor.md](policy-advisor.md) for the full pipeline design. | RPC | Description | |-----|-------------| @@ -258,27 +399,27 @@ These RPCs support the sandbox-initiated policy recommendation pipeline. The san Defined in `proto/inference.proto`, implemented in `crates/openshell-server/src/inference.rs` as `InferenceService`. -The gateway acts as the control plane for inference configuration. It stores a single managed cluster inference route (named `inference.local`) and delivers resolved route bundles to sandbox pods. The gateway does not execute inference requests -- sandboxes connect directly to inference backends using the credentials and endpoints provided in the bundle. +The gateway acts as the control plane for inference configuration. It stores a single managed gateway inference route (named `inference.local`) and delivers resolved route bundles to sandbox pods. The gateway does not execute inference requests -- sandboxes connect directly to inference backends using the credentials and endpoints provided in the bundle. -#### Cluster Inference Configuration +#### Gateway Inference Configuration -The gateway manages a single cluster-wide inference route that maps to a provider record. When set, the route stores only a `provider_name` and `model_id` reference. At bundle resolution time, the gateway looks up the referenced provider and derives the endpoint URL, API key, protocols, and provider type from it. This late-binding design means provider credential rotations are automatically reflected in the next bundle fetch without updating the route itself. +The gateway manages a single gateway-wide inference route that maps to a provider record. When set, the route stores only a `provider_name` and `model_id` reference. At bundle resolution time, the gateway looks up the referenced provider and derives the endpoint URL, API key, protocols, and provider type from it. This late-binding design means provider credential rotations are automatically reflected in the next bundle fetch without updating the route itself. | RPC | Description | |-----|-------------| -| `SetClusterInference` | Configures the cluster inference route. Validates `provider_name` and `model_id` are non-empty, verifies the named provider exists and has a supported type for inference (openai, anthropic, nvidia), validates the provider has a usable API key, then upserts the `inference.local` route record. Increments a monotonic `version` on each update. Returns the configured `provider_name`, `model_id`, and `version`. | -| `GetClusterInference` | Returns the current cluster inference configuration (`provider_name`, `model_id`, `version`). Returns `NotFound` if no cluster inference is configured, or `FailedPrecondition` if the stored route has empty provider/model metadata. | +| `SetClusterInference` | Configures the gateway inference route. Validates `provider_name` and `model_id` are non-empty, verifies the named provider exists and has a supported type for inference (openai, anthropic, nvidia), validates the provider has a usable API key, then upserts the `inference.local` route record. Increments a monotonic `version` on each update. Returns the configured `provider_name`, `model_id`, and `version`. | +| `GetClusterInference` | Returns the current gateway inference configuration (`provider_name`, `model_id`, `version`). Returns `NotFound` if no gateway inference is configured, or `FailedPrecondition` if the stored route has empty provider/model metadata. | | `GetInferenceBundle` | Returns the resolved inference route bundle for sandbox consumption. See [Route Bundle Delivery](#route-bundle-delivery) below. | #### Route Bundle Delivery -The `GetInferenceBundle` RPC resolves the managed cluster route into a `GetInferenceBundleResponse` containing fully materialized route data that sandboxes can use directly. +The `GetInferenceBundle` RPC resolves the managed gateway route into a `GetInferenceBundleResponse` containing fully materialized route data that sandboxes can use directly. The trait method delegates to `resolve_inference_bundle(store)` (`crates/openshell-server/src/inference.rs`), which takes `&Store` instead of `&self`. This extraction decouples bundle resolution from `ServerState`, enabling direct unit testing against an in-memory SQLite store without constructing a full server. The `GetInferenceBundleResponse` includes: -- **`routes`** -- a list of `ResolvedRoute` messages containing base URL, model ID, API key, protocols, and provider type. Currently contains zero or one routes (the managed cluster route). +- **`routes`** -- a list of `ResolvedRoute` messages containing base URL, model ID, API key, protocols, and provider type. Currently contains zero or one routes (the managed gateway route). - **`revision`** -- a hex-encoded hash computed from route contents. Sandboxes compare this value to detect when their route set has changed. - **`generated_at_ms`** -- epoch milliseconds when the bundle was assembled. @@ -315,9 +456,9 @@ The HTTP router (`crates/openshell-server/src/http.rs`) merges two sub-routers: | Path | Method | Response | |------|--------|----------| -| `/connect/ssh` | CONNECT | Upgrades the connection to a bidirectional TCP tunnel to a sandbox pod's SSH port | +| `/connect/ssh` | CONNECT | Upgrades the connection to a bidirectional byte bridge tunneled through `SupervisorSessionRegistry::open_relay` | -See [SSH Tunnel Gateway](#ssh-tunnel-gateway) for details. +See [Connect Flow (SSH Tunnel)](#connect-flow-ssh-tunnel) for details. ### Cloudflare Endpoints @@ -328,7 +469,7 @@ See [SSH Tunnel Gateway](#ssh-tunnel-gateway) for details. ## Watch Sandbox Stream -The `WatchSandbox` RPC (`crates/openshell-server/src/grpc.rs`) provides a multiplexed server-streaming response that can include sandbox status snapshots, gateway log lines, and platform events. +The `WatchSandbox` RPC (`crates/openshell-server/src/grpc/`) provides a multiplexed server-streaming response that can include sandbox status snapshots, gateway log lines, and platform events. ### Request Options @@ -336,7 +477,7 @@ The `WatchSandboxRequest` controls what the stream includes: - `follow_status` -- subscribe to `SandboxWatchBus` notifications and re-read the sandbox record on each change. - `follow_logs` -- subscribe to `TracingLogBus` for gateway log lines correlated by `sandbox_id`. -- `follow_events` -- subscribe to `PlatformEventBus` for Kubernetes events correlated to the sandbox. +- `follow_events` -- subscribe to `PlatformEventBus` for compute-driver platform events correlated to the sandbox. - `log_tail_lines` -- replay the last N log lines before following (default 200). - `stop_on_terminal` -- end the stream when the sandbox reaches the `Ready` phase. Note: `Error` phase does not stop the stream because it may be transient (e.g., `ReconcilerError`). @@ -355,8 +496,8 @@ The `WatchSandboxRequest` controls what the stream includes: ```mermaid graph LR - SW["spawn_sandbox_watcher"] - ET["spawn_kube_event_tailer"] + CW["ComputeRuntime watcher"] + PE["Platform events
(driver watch)"] TL["SandboxLogLayer
(tracing layer)"] WB["SandboxWatchBus
(broadcast per ID)"] @@ -365,9 +506,9 @@ graph LR WS["WatchSandbox stream"] - SW -->|"notify(id)"| WB + CW -->|"notify(id)"| WB TL -->|"publish(id, log_event)"| LB - ET -->|"publish(id, platform_event)"| PB + PE -->|"publish(id, platform_event)"| PB WB -->|"subscribe(id)"| WS LB -->|"subscribe(id)"| WS @@ -375,6 +516,7 @@ graph LR ``` All buses use `tokio::sync::broadcast` channels keyed by sandbox ID. Buffer sizes: + - `SandboxWatchBus`: 128 (signals only, no payload -- just `()`) - `TracingLogBus`: 1024 (full `SandboxStreamEvent` payloads) - `PlatformEventBus`: 1024 (full `SandboxStreamEvent` payloads) @@ -385,52 +527,6 @@ Broadcast lag is translated to `Status::resource_exhausted` via `broadcast_to_st **Validation:** `WatchSandbox` validates that the sandbox exists before subscribing to any bus, preventing entries from being created for non-existent IDs. `PushSandboxLogs` validates sandbox existence once on the first batch of the stream. -## Remote Exec via SSH - -The `ExecSandbox` RPC (`crates/openshell-server/src/grpc.rs`) executes a command inside a sandbox pod over SSH and streams stdout/stderr/exit back to the client. - -### Execution Flow - -1. Validate request: `sandbox_id`, `command`, and environment key format (`^[A-Za-z_][A-Za-z0-9_]*$`). -2. Verify sandbox exists and is in `Ready` phase. -3. Resolve target: prefer agent pod IP, fall back to Kubernetes service DNS (`..svc.cluster.local`). If the sandbox is not connectable yet (for example the pod exists but has no IP), the gateway returns `FAILED_PRECONDITION` instead of surfacing the condition as an internal server fault. -4. Build the remote command string: sort environment variables, shell-escape all values, prepend `cd &&` if `workdir` is set. -5. **Start a single-use SSH proxy**: binds an ephemeral local TCP port, accepts one connection, performs the NSSH1 handshake with the sandbox, and bidirectionally copies data. -6. **Connect via `russh`**: establishes an SSH connection through the local proxy, authenticates with `none` auth as user `sandbox`, opens a session channel, and executes the command. -7. Stream `ExecSandboxStdout`, `ExecSandboxStderr` chunks as they arrive, then send `ExecSandboxExit` with the exit code. -8. On timeout (if `timeout_seconds > 0`), send exit code 124 (matching the `timeout(1)` convention). - -### NSSH1 Handshake Protocol - -The single-use SSH proxy and the SSH tunnel endpoint both use the same handshake: - -``` -NSSH1 \n -``` - -- `token` -- session token or a one-time UUID. -- `timestamp` -- Unix epoch seconds. -- `nonce` -- UUID v4. -- `hmac_signature` -- `HMAC-SHA256(secret, "{token}|{timestamp}|{nonce}")`, hex-encoded. -- Expected response: `OK\n` from the sandbox. - -The `ssh_handshake_skew_secs` configuration controls how much clock skew is tolerated. - -## SSH Tunnel Gateway - -The SSH tunnel endpoint (`crates/openshell-server/src/ssh_tunnel.rs`) allows external SSH clients to reach sandbox pods through the gateway using HTTP CONNECT upgrades. - -### Request Flow - -1. Client sends `CONNECT /connect/ssh` with headers `x-sandbox-id` and `x-sandbox-token`. -2. Handler validates the method is CONNECT, extracts headers. -3. Fetches the `SshSession` from the store by token; rejects if revoked or if `sandbox_id` does not match. -4. Fetches the `Sandbox`; rejects if not in `Ready` phase. -5. Resolves the connect target: agent pod IP if available, otherwise Kubernetes service DNS. -6. Returns `200 OK`, then upgrades the connection via `hyper::upgrade::on()`. -7. In a spawned task: connects to the sandbox's SSH port, performs the NSSH1 handshake, then bidirectionally copies bytes between the upgraded HTTP connection and the sandbox TCP stream. -8. On completion, gracefully shuts down the write-half of the upgraded connection for clean EOF handling. - ## Persistence Layer ### Store Architecture @@ -486,7 +582,7 @@ The `generate_name()` function produces random 6-character lowercase alphabetic ### Deployment Storage -The gateway runs as a Kubernetes **StatefulSet** with a `volumeClaimTemplate` that provisions a 1Gi `ReadWriteOnce` PersistentVolumeClaim mounted at `/var/openshell`. On k3s clusters this uses the built-in `local-path-provisioner` StorageClass (the cluster default). The SQLite database file at `/var/openshell/openshell.db` survives pod restarts and rescheduling. +The gateway runs as a Kubernetes **StatefulSet** with a `volumeClaimTemplate` that provisions a 1Gi `ReadWriteOnce` PersistentVolumeClaim mounted at `/var/openshell`. The cluster's default StorageClass supplies the volume unless an operator customizes the chart. The SQLite database file at `/var/openshell/openshell.db` survives pod restarts and rescheduling. The Helm chart template is at `deploy/helm/openshell/templates/statefulset.yaml`. @@ -496,28 +592,48 @@ The Helm chart template is at `deploy/helm/openshell/templates/statefulset.yaml` - **Get / Delete**: Operate by primary key (`id`), filtered by `object_type`. - **List**: Pages by `limit` + `offset` with deterministic ordering: `ORDER BY created_at_ms ASC, name ASC`. The secondary sort on `name` prevents unstable ordering when rows share the same millisecond timestamp. -## Kubernetes Integration +## Compute Driver Integration -### Sandbox CRD Management +### Kubernetes Driver `KubernetesComputeDriver` (`crates/openshell-driver-kubernetes/src/driver.rs`) manages `agents.x-k8s.io/v1alpha1/Sandbox` CRDs behind the gateway's compute interface. The gateway binds to that driver through `ComputeDriverService` (`crates/openshell-driver-kubernetes/src/grpc.rs`) in-process, so the same `openshell.compute.v1.ComputeDriver` request and response types are exercised whether the driver is invoked locally or served over gRPC. - **Get**: `GetSandbox` looks up a sandbox CRD by name and returns a driver-native platform observation (`openshell.compute.v1.DriverSandbox`) with raw status and condition data from the object. - **List**: `ListSandboxes` enumerates sandbox CRDs and returns driver-native platform observations for each, sorted by name for stable results. -- **Create**: Translates an internal `openshell.compute.v1.DriverSandbox` message into a Kubernetes `DynamicObject` with labels (`openshell.ai/sandbox-id`, `openshell.ai/managed-by: openshell`) and a spec that includes the pod template, environment variables, and gateway-required env vars (`OPENSHELL_SANDBOX_ID`, `OPENSHELL_ENDPOINT`, `OPENSHELL_SSH_LISTEN_ADDR`, etc.). When callers do not provide custom `volumeClaimTemplates`, the driver injects a default `workspace` PVC and mounts it at `/sandbox` so the default sandbox home/workdir survives pod rescheduling. +- **Create**: Translates an internal `openshell.compute.v1.DriverSandbox` message into a Kubernetes `DynamicObject` with labels (`openshell.ai/sandbox-id`, `openshell.ai/managed-by: openshell`) and a spec that includes the pod template, environment variables, and gateway-required env vars (`OPENSHELL_SANDBOX_ID`, `OPENSHELL_ENDPOINT`, `OPENSHELL_SSH_SOCKET_PATH`, etc.). `OPENSHELL_SSH_SOCKET_PATH` is set from the driver's `--sandbox-ssh-socket-path` flag (default `/run/openshell/ssh.sock`) so the in-sandbox SSH daemon binds a Unix socket rather than a TCP port. When callers do not provide custom `volumeClaimTemplates`, the driver injects a default `workspace` PVC and mounts it at `/sandbox` so the default sandbox home/workdir survives pod rescheduling. - **Delete**: Calls the Kubernetes API to delete the CRD by name. Returns `false` if already gone (404). -- **Stop**: `proto/compute_driver.proto` now reserves `StopSandbox` for a non-destructive lifecycle transition. Resume is intentionally not a dedicated compute-driver RPC; the gateway is expected to auto-resume a stopped sandbox when a client connects or executes into it. -- **Pod IP resolution**: `agent_pod_ip()` fetches the agent pod and reads `status.podIP`. +- **Stop**: `proto/compute_driver.proto` reserves `StopSandbox` for a non-destructive lifecycle transition. Resume is intentionally not a dedicated compute-driver RPC; the gateway auto-resumes a stopped sandbox when a client connects or executes into it. -### Sandbox Watcher +The gateway reaches the sandbox exclusively through the supervisor-initiated `ConnectSupervisor` session, so the driver never returns sandbox network endpoints. -The Kubernetes driver emits `WatchSandboxes` events through `proto/compute_driver.proto`. `ComputeRuntime` consumes that stream, translates the driver-native snapshots into public `openshell.v1.Sandbox` resources, derives the public phase, and applies the results to the store. +### Docker Driver -- **Applied**: Extracts the sandbox ID from labels (or falls back to name prefix stripping), reads the CRD status, emits a driver-native snapshot, and lets the gateway translate that into the stored public sandbox record. Notifies the watch bus. -- **Deleted**: Removes the sandbox record from the store and the index. Notifies the watch bus. -- **Restarted**: Re-processes all objects (full resync). +The Docker driver (`crates/openshell-driver-docker/src/lib.rs`) is an in-process compute backend for local standalone gateways. It creates one Docker container per sandbox, labels each container with `openshell.ai/managed-by=openshell`, `openshell.ai/sandbox-id`, `openshell.ai/sandbox-name`, and `openshell.ai/sandbox-namespace`, and bind-mounts a Linux `openshell-sandbox` supervisor binary into the container. -In addition to the watch stream, `ComputeRuntime` periodically calls `ComputeDriver/ListSandboxes` through the in-process `ComputeDriverService` and reconciles the store to that full driver snapshot. Public `GetSandbox` and `ListSandboxes` handlers remain store-backed, but the store is refreshed from the driver on a timer so the gateway still exercises the compute-driver RPC surface for reconciliation. +- **Create**: Pulls or validates the sandbox image according to `sandbox_image_pull_policy`, creates a labeled container, mounts the supervisor binary and optional TLS material, and starts the container with the supervisor as entrypoint. +- **Bridge networking**: Ensures a local Docker bridge network exists (`openshell-docker` by default) and starts every sandbox container on that network instead of using `network_mode=host`. +- **Gateway callback routing**: On native Linux Docker, injects `host.openshell.internal` with the bridge gateway IP and reports that bridge gateway IP plus the normal gateway port to `run_server()` as an extra listener. If the primary listener already binds the wildcard address for that port, the extra address is covered and is not bound a second time. On Docker Desktop, the bridge gateway IP belongs to Docker Desktop's VM rather than the macOS/Windows host, so the driver maps `host.openshell.internal` to Docker's `host-gateway` alias and does not request an extra listener. `OPENSHELL_ENDPOINT` inside Docker sandboxes uses the configured scheme and points at `host.openshell.internal:` in both cases. +- **Environment ownership**: Merges template and spec environment first, then overwrites driver-owned supervisor variables, including `PATH`, `OPENSHELL_ENDPOINT`, `OPENSHELL_SANDBOX_ID`, `OPENSHELL_SSH_SOCKET_PATH`, and `OPENSHELL_SANDBOX_COMMAND`. This keeps privileged supervisor setup from resolving helper binaries through a user-controlled search path. +- **List/Get/Watch**: Reads labeled containers in the configured sandbox namespace and derives driver-native sandbox status from Docker state plus supervisor relay readiness. +- **Stop**: Stops the matching labeled container without deleting it. +- **Delete**: Force-removes the matching labeled container. +- **Gateway shutdown**: On SIGINT or SIGTERM, `run_server()` leaves the accept loop and calls the Docker shutdown cleanup hook. The hook stops all running, restarting, or paused OpenShell-managed containers in the configured sandbox namespace so local sandboxes do not keep running after the gateway exits. +- **Gateway startup resume**: Before the watch and reconcile loops spawn, `ComputeRuntime::resume_persisted_sandboxes()` walks every sandbox record in the store. For each sandbox whose phase is `Provisioning`, `Ready`, or `Unknown`, it asks the Docker driver to start the labeled container if it is in the `exited` or `created` state (`StartupResume::resume_sandbox`). Containers in `running` or `restarting` are left alone; `paused`, `dead`, and `removing` are skipped. If the matching container has disappeared, the sandbox is moved to phase `Error` with reason `BackendResourceMissing`; if the start call fails, the sandbox moves to `Error` with reason `ResumeFailed`. This is what makes sandboxes survive a graceful gateway restart end-to-end: shutdown stops them, the next startup resumes them, and the store remains the source of truth across the cycle. Drivers that do not need this hook (Kubernetes, Podman, VM) leave `startup_resume = None`, which makes the resume sweep a no-op. +- **Handshake secret**: The Docker driver does not inject `OPENSHELL_SSH_HANDSHAKE_SECRET` or `OPENSHELL_SSH_HANDSHAKE_SKEW_SECS` into containers. Supervisor relay auth relies on the gateway connection rather than a Docker-visible container env secret. + +### VM Driver + +`VmDriver` (`crates/openshell-driver-vm/src/driver.rs`) is served by the standalone `openshell-driver-vm` process. The gateway spawns that binary on demand and talks to it over the internal `openshell.compute.v1.ComputeDriver` gRPC contract via a Unix domain socket. + +- **Create**: The VM driver process exports the selected sandbox image from the local Docker daemon, rewrites it into a sandbox-specific guest rootfs, injects an explicitly configured guest mTLS bundle when the gateway callback endpoint is `https://`, then re-execs itself in a hidden helper mode that loads libkrun directly and boots the supervisor. +- **Networking**: The helper starts an embedded `gvproxy`, wires it into libkrun as virtio-net, and gives the guest outbound connectivity. No inbound TCP listener is needed — the supervisor reaches the gateway over its outbound `ConnectSupervisor` stream. +- **Gateway callback**: The guest init script configures `eth0` for gvproxy networking, seeds `/etc/hosts` so `host.openshell.internal` resolves to the gvproxy gateway IP (`192.168.127.1`), preserves gvproxy's legacy `host.containers.internal` / `host.docker.internal` DNS answers, prefers the configured `OPENSHELL_GRPC_ENDPOINT`, and falls back to those aliases or the raw gateway IP when local hostname resolution is unavailable on macOS. +- **Guest boot**: The sandbox guest runs a minimal init script that starts `openshell-sandbox` directly as PID 1 inside the VM. +- **Watch stream**: Emits provisioning, ready, error, deleting, deleted, and platform-event updates so the gateway store remains the durable source of truth. + +### Compute Runtime + +`ComputeRuntime` consumes the driver-native watch stream from `WatchSandboxes`, translates the snapshots into public `openshell.v1.Sandbox` resources, derives the public phase, and applies the results to the store. In parallel, it periodically calls `ListSandboxes` and reconciles the store to the full driver snapshot; public `GetSandbox` and `ListSandboxes` handlers remain store-backed but are refreshed from the driver on a timer. ### Gateway Phase Derivation @@ -534,7 +650,7 @@ In addition to the watch stream, `ComputeRuntime` periodically calls `ComputeDri **Transient reasons** (will retry, stay in `Provisioning`): `ReconcilerError`, `DependenciesNotReady`. All other `Ready=False` reasons are treated as terminal failures (`Error` phase). -### Kubernetes Event Tailer +### Kubernetes Event Correlation The Kubernetes driver also watches namespace-scoped Kubernetes `Event` objects and correlates them to sandbox IDs before emitting them as compute-driver platform events: @@ -551,29 +667,35 @@ Matched events are published to the `PlatformEventBus` as `SandboxStreamEvent::E - `sandbox_name_to_id: HashMap` - `agent_pod_to_id: HashMap` -Updated by the sandbox watcher on every Applied event and by gRPC handlers during sandbox creation. Used by the event tailer to map Kubernetes event objects back to sandbox IDs. +Updated by the compute watcher on every driver observation and by gRPC handlers during sandbox creation. Used by the compute-driver event correlator to map platform events back to sandbox IDs. + +## Observability + +Supervisor session telemetry is currently emitted as plain `tracing` events from `supervisor_session.rs` (accepted, superseded, ended, relay opened/claimed). OCSF structured logging on the gateway side is a tracked follow-up -- the `openshell-ocsf` crate needs a `GatewayContext` equivalent to the sandbox's `SandboxContext` before events like `network.activity` or `app.lifecycle` can be emitted here. Sandbox-side OCSF already covers SSH authentication, network decisions, and supervisor lifecycle. ## Error Handling - **gRPC errors**: All gRPC handlers return `tonic::Status` with appropriate codes: - - `InvalidArgument` for missing/malformed fields - - `NotFound` for nonexistent objects - - `AlreadyExists` for duplicate creation - - `FailedPrecondition` for state violations (e.g., exec on non-Ready sandbox, missing provider) - - `Internal` for store/decode/Kubernetes failures - - `PermissionDenied` for policy violations - - `ResourceExhausted` for broadcast lag (missed messages) - - `Cancelled` for closed broadcast channels - -- **HTTP errors**: The SSH tunnel handler returns HTTP status codes directly (`401`, `404`, `405`, `412`, `500`, `502`). + - `InvalidArgument` for missing/malformed fields, including a non-`Init` first frame on `RelayStream`. + - `NotFound` for nonexistent objects, including unknown or expired relay channels on `claim_relay`. + - `AlreadyExists` for duplicate creation. + - `FailedPrecondition` for state violations (e.g., exec on non-Ready sandbox, missing provider). + - `Unavailable` when the supervisor session for a sandbox is not connected within `open_relay`'s wait window, or when the supervisor's outbound channel has closed between lookup and send. + - `DeadlineExceeded` when a pending relay slot is claimed past `RELAY_PENDING_TIMEOUT`, or when `relay_rx` fails to deliver in time. + - `Internal` for store/decode/driver failures and for the `claim_relay` case where the waiter has dropped the oneshot receiver. + - `PermissionDenied` for policy violations. + - `ResourceExhausted` for broadcast lag (missed messages). + - `Cancelled` for closed broadcast channels. + +- **HTTP errors**: The SSH tunnel handler returns HTTP status codes directly (`401`, `404`, `405`, `412`, `429`, `500`, `502`). `502` indicates the supervisor relay could not be opened; `429` indicates a per-token or per-sandbox concurrent-tunnel limit. - **Connection errors**: Logged at `error` level but do not crash the gateway. TLS handshake failures and individual connection errors are caught and logged per-connection. -- **Background task errors**: The sandbox watcher and event tailer log warnings for individual processing failures but continue running. If the watcher stream ends, it logs a warning and the task exits (no automatic restart). +- **Background task errors**: The compute watcher and relay reaper log warnings for individual processing failures but continue running. If the watcher stream ends, it logs a warning and the task exits (no automatic restart). ## Cross-References -- [Sandbox Architecture](sandbox.md) -- sandbox-side policy enforcement, proxy, and isolation details +- [Sandbox Architecture](sandbox.md) -- sandbox-side policy enforcement, supervisor, and the local SSH daemon on the other end of the relay - [Gateway Settings Channel](gateway-settings.md) -- runtime settings channel, two-tier resolution, CLI/TUI commands - [Inference Routing](inference-routing.md) -- end-to-end inference interception flow, sandbox-side proxy logic, and route resolution - [Container Management](build-containers.md) -- how sandbox container images are built and configured diff --git a/architecture/inference-routing.md b/architecture/inference-routing.md index e2fd671782..01b6a853a0 100644 --- a/architecture/inference-routing.md +++ b/architecture/inference-routing.md @@ -62,7 +62,7 @@ File: `crates/openshell-server/src/inference.rs` The gateway implements the `Inference` gRPC service defined in `proto/inference.proto`. -### Cluster inference set/get +### Gateway inference set/get `SetClusterInference` takes a `provider_name` and `model_id`. It: @@ -73,7 +73,7 @@ The gateway implements the `Inference` gRPC service defined in `proto/inference. 5. Builds a managed route spec that stores only `provider_name` and `model_id`. The spec intentionally leaves `base_url`, `api_key`, and `protocols` empty -- these are resolved dynamically at bundle time from the provider record. 6. Upserts the route with name `inference.local`. Version starts at 1 and increments monotonically on each update. -`GetClusterInference` returns `provider_name`, `model_id`, and `version` for the managed route. Returns `NOT_FOUND` if cluster inference is not configured. +`GetClusterInference` returns `provider_name`, `model_id`, and `version` for the managed route. Returns `NOT_FOUND` if gateway inference is not configured. ### Bundle delivery @@ -87,7 +87,7 @@ The gateway implements the `Inference` gRPC service defined in `proto/inference. Because resolution happens at request time, credential rotation and endpoint changes on the provider record take effect on the next bundle fetch without re-running `SetClusterInference`. -An empty route list is valid and indicates cluster inference is not yet configured. +An empty route list is valid and indicates gateway inference is not yet configured. ### Proto definitions @@ -109,7 +109,7 @@ Files: - `crates/openshell-sandbox/src/lib.rs` -- inference context initialization, route refresh - `crates/openshell-sandbox/src/grpc_client.rs` -- `fetch_inference_bundle()` -In cluster mode, the sandbox starts a background refresh loop as soon as the inference context is created. The loop polls the gateway every 5 seconds by default (`OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS` override) and uses the bundle revision hash to skip no-op cache writes. The revision hash covers all route fields including `timeout_secs`, so any configuration change (provider, model, or timeout) triggers a cache update on the next poll. +In gateway bundle mode, the sandbox starts a background refresh loop as soon as the inference context is created. The loop polls the gateway every 5 seconds by default (`OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS` override) and uses the bundle revision hash to skip no-op cache writes. The revision hash covers all route fields including `timeout_secs`, so any configuration change (provider, model, or timeout) triggers a cache update on the next poll. ### Interception flow @@ -146,9 +146,9 @@ If no pattern matches, the proxy returns `403 Forbidden` with `{"error": "connec ### Route cache - `InferenceContext` holds a `Router`, the pattern list, and an `Arc>>` route cache. -- In cluster mode, `spawn_route_refresh()` polls `GetInferenceBundle` every 5 seconds (`OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS`). On failure, stale routes are kept. +- In gateway bundle mode, `spawn_route_refresh()` polls `GetInferenceBundle` every 5 seconds (`OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS`). On failure, stale routes are kept. - In file mode (`--inference-routes`), routes load once at startup from YAML. No refresh task is spawned. -- In cluster mode, an empty initial bundle still enables the inference context so the refresh task can pick up later configuration. +- In gateway bundle mode, an empty initial bundle still enables the inference context so the refresh task can pick up later configuration. ### Bundle-to-route conversion @@ -171,16 +171,17 @@ Files: `prepare_backend_request()` (shared by both buffered and streaming paths) rewrites outgoing requests: 1. **Auth injection**: Uses the route's `AuthHeader` -- either `Authorization: Bearer ` or a custom header (e.g. `x-api-key: ` for Anthropic). -2. **Header stripping**: Removes `authorization`, `x-api-key`, `host`, and any header names that will be set from route defaults. -3. **Default headers**: Applies route-level default headers (e.g. `anthropic-version: 2023-06-01`) unless the client already sent them. -4. **Model rewrite**: Parses the request body as JSON and replaces the `model` field with the route's configured model. Non-JSON bodies are forwarded unchanged. -5. **URL construction**: `build_backend_url()` appends the request path to the route endpoint. If the endpoint already ends with `/v1` and the request path starts with `/v1/`, the duplicate prefix is deduplicated. +2. **Header allowlist**: Keeps only explicitly approved request headers: common inference headers (`content-type`, `accept`, `accept-encoding`, `user-agent`), route-specific passthrough headers (for example `openai-organization`, `x-model-id`, `anthropic-version`, `anthropic-beta`), and any route default header names. +3. **Header stripping**: Removes `authorization`, `x-api-key`, `host`, `content-length`, hop-by-hop headers, and any non-allowlisted request headers. +4. **Default headers**: Applies route-level default headers (e.g. `anthropic-version: 2023-06-01`) unless the client already sent them. +5. **Model rewrite**: Parses the request body as JSON and replaces the `model` field with the route's configured model. Non-JSON bodies are forwarded unchanged. +6. **URL construction**: `build_backend_url()` appends the request path to the route endpoint. If the endpoint already ends with `/v1` and the request path starts with `/v1/`, the duplicate prefix is deduplicated. ### Header sanitization -Before forwarding inference requests, the proxy strips sensitive and hop-by-hop headers from both requests and responses: +Before forwarding inference requests, the router enforces a route-aware request allowlist and strips sensitive/framing headers. Response sanitization remains framing-only: -- **Request**: `authorization`, `x-api-key`, `host`, `content-length`, and hop-by-hop headers (`connection`, `keep-alive`, `proxy-authenticate`, `proxy-authorization`, `proxy-connection`, `te`, `trailer`, `transfer-encoding`, `upgrade`). +- **Request**: forwards only common inference headers plus route-specific passthrough headers and route default header names. Always strips `authorization`, `x-api-key`, `host`, `content-length`, unknown headers such as `cookie`, and hop-by-hop headers (`connection`, `keep-alive`, `proxy-authenticate`, `proxy-authorization`, `proxy-connection`, `te`, `trailer`, `transfer-encoding`, `upgrade`). - **Response**: `content-length` and hop-by-hop headers. ### Response streaming @@ -213,7 +214,7 @@ This eliminates full-body buffering for streaming responses (SSE). Time-to-first When the proxy truncates a streaming response, it injects an SSE error event via `format_sse_error()` (in `crates/openshell-sandbox/src/l7/inference.rs`) before sending the HTTP chunked terminator: -``` +```text data: {"error":{"message":"","type":"proxy_stream_error"}} ``` @@ -279,7 +280,7 @@ Validation at load time requires either `api_key` or `api_key_env` to resolve, a | Status | Condition | |--------|-----------| | `403` | Request on `inference.local` does not match a recognized inference API pattern | -| `503` | Pattern matched but route cache is empty (cluster inference not configured) | +| `503` | Pattern matched but route cache is empty (gateway inference not configured) | | `400` | No compatible route for the detected source protocol | | `401` | Upstream returned unauthorized | | `502` | Upstream protocol error or internal router error | @@ -327,15 +328,15 @@ The system route is stored as a separate `InferenceRoute` record in the gateway ## CLI Surface -Cluster inference commands: +Gateway inference commands: -- `openshell inference set --provider --model [--timeout ]` -- configures user-facing cluster inference +- `openshell inference set --provider --model [--timeout ]` -- configures user-facing gateway inference - `openshell inference set --system --provider --model [--timeout ]` -- configures system inference - `openshell inference update [--provider ] [--model ] [--timeout ]` -- updates individual fields without resetting others - `openshell inference get` -- displays both user and system inference configuration - `openshell inference get --system` -- displays only the system inference configuration -The `--provider` flag references a provider record name (not a provider type). The provider must already exist in the cluster and have a supported inference type (`openai`, `anthropic`, or `nvidia`). +The `--provider` flag references a provider record name (not a provider type). The provider must already exist on the gateway and have a supported inference type (`openai`, `anthropic`, or `nvidia`). The `--timeout` flag sets the per-request timeout in seconds for upstream inference calls. When omitted or set to `0`, the default of 60 seconds applies. Timeout changes propagate to running sandboxes within the route refresh interval (5 seconds by default). diff --git a/architecture/object-metadata.md b/architecture/object-metadata.md new file mode 100644 index 0000000000..961576e164 --- /dev/null +++ b/architecture/object-metadata.md @@ -0,0 +1,426 @@ +# Object Metadata Convention + +## Overview + +OpenShell adopts a Kubernetes-style object metadata convention for all top-level domain objects. This standardizes how resources are identified, labeled, and queried across the platform. All resources that users interact with directly (Sandbox, Provider, SshSession, InferenceRoute) follow this convention. + +## Core Principles + +### 1. Uniform Metadata Structure + +All top-level objects embed a common `ObjectMeta` message containing: + +- **Stable ID**: Server-generated UUID that never changes +- **Human-readable name**: User-friendly identifier (unique per object type) +- **Creation timestamp**: Milliseconds since Unix epoch +- **Labels**: Key-value pairs for filtering and organization + +### 2. Trait-Based Access + +Rather than accessing metadata fields directly (e.g., `sandbox.metadata.as_ref().unwrap().id`), code uses trait methods from `openshell_core::metadata`: + +```rust +use openshell_core::{ObjectId, ObjectName, ObjectLabels}; + +let id = sandbox.object_id(); // Returns &str +let name = sandbox.object_name(); // Returns &str +let labels = sandbox.object_labels(); // Returns Option> +``` + +This provides: + +- **Uniform API** across all object types +- **Graceful fallback** (returns empty string if metadata is None) +- **Reduced boilerplate** in code that works with multiple object types + +### 3. Labels for Organization and Filtering + +Labels are key-value metadata attached to objects for: + +- **Grouping** related resources (e.g., all dev environment sandboxes) +- **Filtering** in list operations (e.g., show only sandboxes with `team=backend`) +- **Automation** and selection in scripts + +## Implementation Pattern + +### Protobuf Definition + +Define `ObjectMeta` once in `proto/datamodel.proto`: + +```protobuf +message ObjectMeta { + string id = 1; + string name = 2; + int64 created_at_ms = 3; + map labels = 4; +} +``` + +Embed it in top-level objects: + +```protobuf +message Sandbox { + ObjectMeta metadata = 1; + SandboxSpec spec = 2; + SandboxStatus status = 3; + int32 phase = 4; + int32 current_policy_version = 5; +} +``` + +**Migration**: When adding metadata to an existing object, shift field numbers to make room for `metadata = 1`. This maintains backward compatibility if done before release. + +### Trait Implementation + +Implement the three traits for each object in `crates/openshell-core/src/metadata.rs`: + +```rust +impl ObjectId for Sandbox { + fn object_id(&self) -> &str { + self.metadata.as_ref().map(|m| m.id.as_str()).unwrap_or("") + } +} + +impl ObjectName for Sandbox { + fn object_name(&self) -> &str { + self.metadata.as_ref().map(|m| m.name.as_str()).unwrap_or("") + } +} + +impl ObjectLabels for Sandbox { + fn object_labels(&self) -> Option> { + self.metadata.as_ref().map(|m| m.labels.clone()) + } +} +``` + +**Pattern**: Always return empty string for missing metadata rather than panicking. This makes code resilient to malformed data. + +### Persistence Layer + +The `Store` trait in `crates/openshell-server/src/persistence/mod.rs` provides three methods for working with objects: + +```rust +// Store/retrieve by stable ID +async fn put_message( + &self, + message: &T, +) -> Result<(), String>; + +async fn get(&self, object_type: &str, id: &str) + -> Result, String>; + +// Retrieve by human-readable name +async fn get_message_by_name( + &self, + name: &str, +) -> Result, String>; +``` + +**Database schema pattern**: Each object type has: + +- `id` column (TEXT PRIMARY KEY) — stable UUID +- `name` column (TEXT UNIQUE NOT NULL) — user-facing name +- `payload` column (BLOB) — serialized protobuf +- `created_at_ms` column (INTEGER) — denormalized from metadata for indexing +- `updated_at_ms` column (INTEGER) — last modification time + +### Label Filtering + +Label selectors follow Kubernetes conventions: + +**Format**: `key1=value1,key2=value2` (comma-separated, AND logic) + +**Implementation**: + +1. Parse selector into key-value pairs +2. For each object, check that ALL selector labels match +3. Return only objects where every label in the selector exists with the exact value + +**SQL pattern** (PostgreSQL with JSONB): + +```sql +WHERE labels @> '{"env": "dev", "team": "backend"}'::jsonb +``` + +**SQL pattern** (SQLite): + +```sql +WHERE json_extract(labels, '$.env') = 'dev' + AND json_extract(labels, '$.team') = 'backend' +``` + +The `list_with_selector` method on `Store` handles this transparently. + +### Validation Rules + +Labels must follow Kubernetes naming conventions (enforced in `crates/openshell-server/src/grpc/validation.rs`): + +**Label keys**: + +- Optional prefix + `/` + name (e.g., `example.com/app` or `app`) +- Prefix: DNS subdomain (lowercase alphanumeric, `-`, `.`, max 253 chars) +- Name: alphanumeric + `-`, `_`, `.`, max 63 chars +- Cannot start or end with `-` or `.` + +**Label values**: + +- Alphanumeric + `-`, `_`, `.` +- Max 63 characters +- Can be empty string + +**Validation functions**: + +```rust +validate_label_key(key: &str) -> Result<(), Status> +validate_label_value(value: &str) -> Result<(), Status> +validate_labels(labels: &HashMap) -> Result<(), Status> +``` + +**Validation timing**: Validate at API ingress (gRPC handlers) before persisting. Reject invalid labels immediately rather than storing and failing later. + +## CLI Integration + +### Creating Objects with Labels + +```bash +openshell sandbox create --label env=dev --label team=backend +openshell provider create openai --label project=research +``` + +**Pattern**: Repeatable `--label key=value` flags parsed into `HashMap`. + +### Listing with Selectors + +```bash +openshell sandbox list --selector env=dev +openshell sandbox list --selector env=dev,team=backend +``` + +**Display**: Show labels in tabular output when present, or in detail views. + +## Testing Requirements + +### Unit Tests + +Test validation logic for: + +- Valid label keys (with and without prefix) +- Invalid keys (bad characters, too long, empty segments) +- Valid label values +- Invalid values (non-alphanumeric, too long) +- Selector parsing + +### Integration Tests + +Test persistence layer: + +- Store object with labels +- Retrieve by name and verify labels present +- Filter with single-label selector +- Filter with multi-label selector (AND logic) +- Empty results for non-matching selector + +### E2E Tests + +Test full workflow through CLI: + +- Create multiple objects with different labels +- List all objects +- Filter by single label +- Filter by multiple labels +- Verify labels persist across gateway restarts + +**Location**: `e2e/rust/tests/sandbox_labels.rs` (or equivalent for each object type) + +## Migration Checklist + +When adding object metadata to a new resource type: + +1. **Proto changes**: + - [ ] Add `ObjectMeta metadata = 1;` field + - [ ] Shift existing field numbers if needed + - [ ] Update any references to old id/name fields + +2. **Trait implementations**: + - [ ] Implement `ObjectId` trait + - [ ] Implement `ObjectName` trait + - [ ] Implement `ObjectLabels` trait + - [ ] Add to `crates/openshell-core/src/metadata.rs` + +3. **Persistence**: + - [ ] Add database migration (SQLite + PostgreSQL) + - [ ] Create `labels` column (JSON/JSONB type) + - [ ] Migrate existing `id`/`name` to `ObjectMeta` + - [ ] Update `ObjectType` implementation + - [ ] Update create/read operations to use new structure + +4. **Validation**: + - [ ] Add label validation in gRPC handlers + - [ ] Validate on create and update operations + - [ ] Test validation with unit tests + +5. **API updates**: + - [ ] Add `label_selector` parameter to List RPC + - [ ] Implement selector filtering in persistence layer + - [ ] Add `labels` field to Create/Update RPCs + +6. **CLI updates**: + - [ ] Add `--label` flag to create command + - [ ] Add `--selector` flag to list command + - [ ] Update completion for label keys (if applicable) + - [ ] Display labels in list and get output + +7. **Tests**: + - [ ] Unit tests for validation + - [ ] Integration tests for persistence + - [ ] E2E tests for CLI workflow + +8. **Documentation**: + - [ ] Update user-facing docs for new flags + - [ ] Add examples with labels to guides + +## Common Patterns + +### Creating Objects with Metadata + +```rust +use crate::persistence::current_time_ms; + +let now_ms = current_time_ms() + .map_err(|e| Status::internal(format!("get current time: {e}")))?; + +let sandbox = Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: uuid::Uuid::new_v4().to_string(), + name: user_provided_name, + created_at_ms: now_ms, + labels: request.labels, + }), + spec: Some(spec), + status: None, + phase: SandboxPhase::Provisioning as i32, + current_policy_version: 0, +}; + +// Validate before persisting +validate_object_metadata(sandbox.metadata.as_ref(), "sandbox")?; +store.put_message(&sandbox).await?; +``` + +### Filtering by Labels + +```rust +let sandboxes = if request.label_selector.is_empty() { + store.list(Sandbox::object_type(), limit, offset).await? +} else { + validate_label_selector(&request.label_selector)?; + store.list_with_selector( + Sandbox::object_type(), + &request.label_selector, + limit, + offset, + ).await? +}; +``` + +### Accessing Metadata Fields + +```rust +use openshell_core::{ObjectId, ObjectName}; + +// Good: trait-based access +let sandbox_id = sandbox.object_id(); +let sandbox_name = sandbox.object_name(); + +// Avoid: direct field access +let sandbox_id = sandbox.metadata.as_ref().unwrap().id.as_str(); // Don't do this +``` + +## Anti-Patterns to Avoid + +### ❌ Bypassing Validation + +```rust +// Bad: storing labels without validation +store.put_message(&sandbox).await?; +``` + +```rust +// Good: validate before storing +validate_labels(&sandbox.metadata.as_ref().unwrap().labels)?; +store.put_message(&sandbox).await?; +``` + +### ❌ Direct Field Access + +```rust +// Bad: fragile to missing metadata +let id = sandbox.metadata.as_ref().unwrap().id.clone(); +``` + +```rust +// Good: trait-based with fallback +let id = sandbox.object_id().to_string(); +``` + +### ❌ Inconsistent Object Construction + +```rust +// Bad: forgetting created_at_ms or labels +let sandbox = Sandbox { + metadata: Some(ObjectMeta { + id: uuid::Uuid::new_v4().to_string(), + name: "test".to_string(), + ..Default::default() // Silently sets created_at_ms=0, labels=empty + }), + ..Default::default() +}; +``` + +```rust +// Good: explicit fields +let sandbox = Sandbox { + metadata: Some(ObjectMeta { + id: uuid::Uuid::new_v4().to_string(), + name: "test".to_string(), + created_at_ms: current_time_ms()?, + labels: request.labels, + }), + ..Default::default() +}; +``` + +### ❌ Client-Side ID Generation + +```rust +// Bad: letting clients specify IDs +let sandbox = Sandbox { + metadata: Some(ObjectMeta { + id: request.id, // Never trust client-provided IDs + .. + }), + .. +}; +``` + +```rust +// Good: server generates stable IDs +let sandbox = Sandbox { + metadata: Some(ObjectMeta { + id: uuid::Uuid::new_v4().to_string(), + .. + }), + .. +}; +``` + +## References + +- **Kubernetes API Conventions**: https://kubernetes.io/docs/reference/using-api/api-concepts/ +- **Label Syntax**: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ +- **Proto definition**: `proto/datamodel.proto` +- **Trait implementations**: `crates/openshell-core/src/metadata.rs` +- **Persistence layer**: `crates/openshell-server/src/persistence/mod.rs` +- **Validation logic**: `crates/openshell-server/src/grpc/validation.rs` +- **E2E tests**: `e2e/rust/tests/sandbox_labels.rs` diff --git a/architecture/oidc-auth.md b/architecture/oidc-auth.md new file mode 100644 index 0000000000..746a6a5192 --- /dev/null +++ b/architecture/oidc-auth.md @@ -0,0 +1,542 @@ +# OIDC Authentication + +OpenShell supports OAuth2/OIDC (OpenID Connect) as an authentication mode alongside mTLS and Cloudflare Access. When enabled, the gateway server validates JWT bearer tokens on gRPC requests against an OIDC provider's JWKS endpoint. The CLI acquires tokens via browser-based login (Authorization Code + PKCE) or environment variables (Client Credentials). + +## Architecture + +```mermaid +graph LR + CLI -->|"Bearer token
(gRPC metadata)"| Gateway["Gateway
Server"] + Gateway -->|response| CLI + Gateway -->|"JWKS (cached)"| Keycloak["Keycloak /
OIDC Provider"] + Browser -->|"Auth Code + PKCE"| Gateway + Keycloak -->|"Token exchange"| CLI + Keycloak -->|"Token exchange"| Browser +``` + +## Auth Modes + +OpenShell determines the authentication strategy per gateway via the `auth_mode` field in gateway metadata (`~/.config/openshell/gateways//metadata.json`): + +| `auth_mode` | Transport | Identity | Token Storage | +|---|---|---|---| +| `"mtls"` | mTLS client cert | Cert CN | N/A | +| `"plaintext"` | HTTP (no TLS) | None | N/A | +| `"cloudflare_jwt"` | Edge TLS (CF Tunnel) | CF Access JWT | `edge_token` file | +| `"oidc"` | mTLS or plaintext | OIDC JWT | `oidc_token.json` | + +## Token Acquisition + +### Interactive: Authorization Code + PKCE + +Used by `openshell gateway login` for interactive CLI sessions. The login flow accepts a `client_id` (the OIDC client application) and an optional `audience` (the API resource server). When `audience` differs from `client_id` — common with providers like Entra ID — it is appended to the authorization URL so the issued token targets the correct API. + +``` +CLI Browser Keycloak + | | | + | 1. Discover OIDC endpoints | | + | GET {issuer}/.well-known/openid-configuration | + | | | + | 2. Generate PKCE pair | | + | code_verifier = random(32 bytes) -> base64url | + | code_challenge = base64url(SHA256(code_verifier)) | + | state = random(16 bytes) -> hex | + | | | + | 3. Start localhost callback | | + | on 127.0.0.1: | | + | | | + | 4. Open browser | | + | -------xdg-open------------->| | + | | 5. Redirect to Keycloak | + | | /auth?response_type=code | + | | &client_id={client_id} | + | | &redirect_uri=localhost:... | + | | &code_challenge=... | + | | &code_challenge_method=S256 | + | | &state=... | + | | [&audience={audience}] | + | | --------------------------->| + | | | + | | 6. User logs | + | | in | + | | | + | | 7. Redirect back | + | | <-- ?code=...&state=... ---| + | | | + | 8. Receive code on callback | | + | <----GET /callback?code=..---| | + | | | + | 9. Validate state matches | | + | | | + | 10. Exchange code for tokens | | + | POST {token_endpoint} | | + | grant_type=authorization_code | + | code=... | | + | redirect_uri=... | | + | client_id={client_id} | | + | code_verifier=... ------------------------------------->| + | | | + | <-- { access_token, refresh_token, expires_in } -----------| + | | | + | 11. Store token bundle | | + | ~/.config/openshell/gateways//oidc_token.json | +``` + +### Non-Interactive: Client Credentials + +Used for CI/automation when `OPENSHELL_OIDC_CLIENT_SECRET` is set. The optional `audience` parameter is included when the API resource server differs from the client ID. + +``` +CI Agent Keycloak + | | + | POST {token_endpoint} | + | grant_type=client_credentials | + | client_id={client_id} | + | client_secret={OPENSHELL_OIDC_CLIENT_SECRET} | + | [audience={audience}] --------------------------------->| + | | + | <-- { access_token, expires_in } -------------------------| + | | + | Store token bundle (no refresh_token) | +``` + +## Token Storage + +OIDC tokens are stored as JSON at `~/.config/openshell/gateways//oidc_token.json` with `0600` permissions: + +```json +{ + "access_token": "eyJhbGci...", + "refresh_token": "eyJhbGci...", + "expires_at": 1718400300, + "issuer": "http://localhost:8180/realms/openshell", + "client_id": "openshell-cli" +} +``` + +The CLI checks `expires_at` before each request. If the token is within 30 seconds of expiry and a `refresh_token` is available, it silently refreshes via the token endpoint's `refresh_token` grant. If refresh fails, the user is prompted to re-authenticate with `openshell gateway login`. + +## Per-Request Flow + +On every gRPC call, the CLI interceptor injects the token as a standard HTTP header: + +``` +authorization: Bearer eyJhbGci... +``` + +The server-side auth middleware (`AuthGrpcRouter` in `multiplex.rs`) classifies each request into one of three categories and processes it accordingly: + +1. **Strip internal markers** — remove `x-openshell-auth-source` from incoming headers to prevent spoofing. +2. **Unauthenticated?** — health probes and reflection pass through with no auth. +3. **Sandbox-secret?** — supervisor RPCs validate the `x-sandbox-secret` header against the server's SSH handshake secret. On success, mark the request with an internal `x-openshell-auth-source: sandbox-secret` header for downstream authorization. +4. **Dual-auth?** — methods like `UpdateConfig` try sandbox-secret first; if no valid secret, fall through to Bearer token validation. +5. **Bearer token** — extract `authorization: Bearer `, decode the JWT header for `kid`, look up the signing key in the JWKS cache, and validate signature (RS256), `exp`, `iss`, `aud` claims. +6. **Authorize** — on successful authentication, check RBAC roles via `AuthzPolicy` (in `authz.rs`). +7. On any failure, return `UNAUTHENTICATED` or `PERMISSION_DENIED` status. + +## JWKS Key Caching + +The server fetches the OIDC provider's JSON Web Key Set at startup via discovery: + +``` +GET {issuer}/.well-known/openid-configuration -> jwks_uri +GET {jwks_uri} -> { keys: [...] } +``` + +Keys are cached in memory with a configurable TTL (default: 1 hour). A `refresh_mutex` serializes refresh operations so concurrent requests coalesce into a single HTTP fetch. The cache refreshes: + +- When the TTL expires (on next request, re-checked under the mutex to avoid thundering herd). +- Immediately when a JWT references a `kid` not in the cache (handles key rotation). + +## Method Authentication Categories + +Every gRPC method falls into one of three categories, defined in `oidc.rs`: + +### Unauthenticated + +These methods require no authentication at all — health probes and infrastructure endpoints. + +| Method / Prefix | Reason | +|---|---| +| `OpenShell/Health` | Kubernetes liveness/readiness probes | +| `Inference/Health` | Inference service health probes | +| `/grpc.reflection.*` | gRPC server reflection (debugging tools) | +| `/grpc.health.*` | gRPC health check protocol | + +### Sandbox-Secret Authenticated + +Sandbox-to-server RPCs authenticate via the `x-sandbox-secret` metadata header, which must match the server's SSH handshake secret. These methods do not use OIDC Bearer tokens. + +| Method | Purpose | +|---|---| +| `SandboxService/GetSandboxConfig` | Supervisor fetches sandbox configuration | +| `ReportPolicyStatus` | Supervisor reports policy enforcement status | +| `PushSandboxLogs` | Supervisor streams sandbox logs to gateway | +| `GetSandboxProviderEnvironment` | Supervisor fetches provider credentials | +| `SubmitPolicyAnalysis` | Supervisor submits policy analysis results | +| `Inference/GetInferenceBundle` | Supervisor fetches resolved inference routes and provider API keys | + +### Dual-Auth + +These methods accept either an OIDC Bearer token (CLI users) or a sandbox secret (supervisor). The middleware tries sandbox-secret first; if not present, it falls through to Bearer token validation. + +| Method | Purpose | +|---|---| +| `UpdateConfig` | Policy and settings mutations | +| `OpenShell/GetSandboxConfig` | CLI reads effective sandbox policy and settings; sandbox callers may still use the shared secret | + +**Sandbox-secret restriction on `UpdateConfig`:** When a sandbox-secret-authenticated caller invokes `UpdateConfig`, the handler in `policy.rs` enforces strict scope limits via `validate_sandbox_secret_update()`. The caller: + +- **Must** provide a sandbox `name` (sandbox-scoped only). +- **Must** include a `policy` payload (policy sync only). +- **May not** set `global = true` (no global config mutation). +- **May not** set `delete_setting` (no setting deletion). +- **May not** provide a `setting_key` (no setting mutation). + +This ensures the sandbox supervisor can sync its own policy on startup but cannot modify global configuration or sandbox settings. + +## Role-Based Access Control (RBAC) + +After JWT validation, the server checks the user's roles against a per-method requirement. Roles are extracted from a configurable claim path in the JWT. + +### Role Mapping + +| Operation | Required Role | +|---|---| +| Health probes, reflection | (no auth — unauthenticated) | +| Supervisor-only RPCs (`SandboxService/GetSandboxConfig`, `GetInferenceBundle`, etc.) | (sandbox secret — no RBAC) | +| UpdateConfig via sandbox secret | (sandbox secret — scope-restricted, no RBAC) | +| OpenShell/GetSandboxConfig via Bearer | user role | +| Sandbox create, list, delete, exec, SSH | user role | +| Provider list, get | user role | +| Provider create, update, delete | admin role | +| Global config/policy updates | admin role | +| Draft policy approvals/rejections | admin role | +| All other authenticated RPCs | user role | + +### Configurable Roles + +The roles claim path and role names are configurable to support different OIDC providers. Each provider stores roles differently in the JWT: + +| Provider | Roles Claim | Example Admin Role | Example User Role | +|---|---|---|---| +| Keycloak | `realm_access.roles` (default) | `openshell-admin` | `openshell-user` | +| Microsoft Entra ID | `roles` | `OpenShell.Admin` | `OpenShell.User` | +| Okta | `groups` | `openshell-admin` | `openshell-user` | +| GitHub | N/A | (empty — skip RBAC) | (empty — skip RBAC) | + +When both `--oidc-admin-role` and `--oidc-user-role` are set to empty strings, RBAC is skipped entirely — any valid JWT is authorized. This supports providers like GitHub that don't emit roles in JWTs (authentication-only mode). + +**Security note on authentication-only mode:** In this mode, the server validates token signature, issuer, and audience, but does not restrict which principals can call which methods. Any entity able to mint a valid token for the configured audience gains full access. For GitHub Actions, this means any workflow in any repository that can request a token with the configured audience is authorized. Consider using scope enforcement (`--oidc-scopes-claim`) or restricting the audience to limit the blast radius. + +## Scope-Based Fine-Grained Permissions + +Scopes provide opt-in, per-method access control on top of roles. When `--oidc-scopes-claim` is set, the server extracts scopes from the JWT and checks them against an exhaustive method-to-scope map. A caller must have both the required role AND the required scope. + +### Scope Definitions + +| Scope | Operations | +|---|---| +| `sandbox:read` | GetSandbox, ListSandboxes, WatchSandbox, GetSandboxLogs, GetSandboxPolicyStatus, ListSandboxPolicies | +| `sandbox:write` | CreateSandbox, DeleteSandbox, ExecSandbox, CreateSshSession, RevokeSshSession | +| `provider:read` | GetProvider, ListProviders | +| `provider:write` | CreateProvider, UpdateProvider, DeleteProvider | +| `config:read` | GetGatewayConfig, GetSandboxConfig, GetDraftPolicy, GetDraftHistory | +| `config:write` | UpdateConfig (Bearer), ApproveDraftChunk, ApproveAllDraftChunks, RejectDraftChunk, EditDraftChunk, UndoDraftChunk, ClearDraftChunks | +| `inference:read` | GetClusterInference | +| `inference:write` | SetClusterInference | +| `openshell:all` | All of the above (wildcard) | + +Methods not listed in the scope map require `openshell:all`. Scopes cannot escalate privilege — `openshell:all` on a user-role token still cannot call admin methods. + +### Authorization Flow + +``` +Request arrives (Bearer-authenticated) + │ + ├── Role check (existing) + │ └── Does identity have required role? No → PERMISSION_DENIED + │ + └── Scope check (only if --oidc-scopes-claim is configured) + ├── Does identity have openshell:all? → proceed + ├── Does identity have required scope for this method? → proceed + └── No → PERMISSION_DENIED("scope 'X' required") +``` + +When `--oidc-scopes-claim` is not set (default), scope enforcement is disabled and roles alone determine access. Auth-only mode (empty role names) still enforces scopes when enabled. + +### Scope Extraction + +The server extracts scopes from the JWT claim path configured by `--oidc-scopes-claim`. Two formats are supported: + +- **Space-delimited string** (Keycloak, Entra ID): `"openid sandbox:read sandbox:write"` +- **JSON array** (Okta): `["sandbox:read", "sandbox:write"]` + +Standard OIDC scopes (`openid`, `profile`, `email`, `offline_access`) are filtered out before enforcement. + +### CLI Scope Requests + +The `--oidc-scopes` flag on `gateway add` and `gateway start` is stored in gateway metadata and included in OAuth2 token requests: + +- **Browser flow**: appended to the `scope` parameter alongside `openid` +- **Client credentials flow**: sent as-is (without `openid`, which is inappropriate for service tokens) +- **Token refresh**: scopes are not re-sent; the authorization server preserves them per RFC 6749 §6 + +### Provider Compatibility + +| Provider | Scopes Claim | Format | Fine-Grained Selection | +|---|---|---|---| +| Keycloak | `scope` | Space-delimited | Yes — client requests specific scopes | +| Okta | `scp` | JSON array | Yes — client requests specific scopes | +| Entra ID | `scp` | Space-delimited | Limited — uses `.default` for all granted permissions | +| GitHub | N/A | N/A | No — use with scopes disabled | + +### Keycloak Client Scopes + +The dev realm (`scripts/keycloak-realm.json`) includes all 9 OpenShell scopes as **optional scopes** on `openshell-cli` and `openshell:all` as a **default scope** on `openshell-ci`. Built-in Keycloak scopes (`openid`, `profile`, `email`, `roles`, `web-origins`, `acr`) are assigned as default scopes on both clients so roles and profile claims are always present regardless of optional scope requests. + +## Server Configuration + +### Server Binary Flags + +These flags configure JWT validation on the `openshell-server` binary: + +| Flag | Env Var | Default | Description | +|---|---|---|---| +| `--oidc-issuer` | `OPENSHELL_OIDC_ISSUER` | (none) | OIDC issuer URL (enables JWT validation) | +| `--oidc-audience` | `OPENSHELL_OIDC_AUDIENCE` | `openshell-cli` | Expected `aud` claim in validated JWTs | +| `--oidc-jwks-ttl` | `OPENSHELL_OIDC_JWKS_TTL` | `3600` | JWKS cache TTL in seconds | +| `--oidc-roles-claim` | `OPENSHELL_OIDC_ROLES_CLAIM` | `realm_access.roles` | Dot-separated path to roles array in JWT | +| `--oidc-admin-role` | `OPENSHELL_OIDC_ADMIN_ROLE` | `openshell-admin` | Role name for admin access | +| `--oidc-user-role` | `OPENSHELL_OIDC_USER_ROLE` | `openshell-user` | Role name for user access | +| `--oidc-scopes-claim` | `OPENSHELL_OIDC_SCOPES_CLAIM` | (empty) | Claim path for scopes; enables scope enforcement when set | + +When `--oidc-issuer` is not set, OIDC validation is disabled and the server falls back to mTLS-only or plaintext behavior. + +### Gateway Start Flags (CLI) + +The `openshell gateway start` command exposes flags that configure both the server and the local gateway metadata: + +| Flag | Default | Description | +|---|---|---| +| `--oidc-issuer` | (none) | OIDC issuer URL; passed to the server binary | +| `--oidc-audience` | `openshell-cli` | Expected `aud` claim; passed to the server binary | +| `--oidc-client-id` | `openshell-cli` | Client ID stored in gateway metadata for CLI login flows | +| `--oidc-roles-claim` | (none) | Passed to the server binary if set | +| `--oidc-admin-role` | (none) | Passed to the server binary if set | +| `--oidc-user-role` | (none) | Passed to the server binary if set | +| `--oidc-scopes-claim` | (none) | Passed to the server binary; enables scope enforcement | +| `--oidc-scopes` | (none) | Stored in gateway metadata; included in CLI token requests | + +The `--oidc-client-id` flag is **not** a server flag — it is stored in gateway metadata and used by the CLI during login. The `--oidc-audience` flag is both a server flag (for JWT validation) and stored in metadata (for token requests). + +### Helm Values + +```yaml +server: + oidc: + issuer: "https://keycloak.example.com/realms/openshell" + audience: "openshell-cli" + jwksTtl: 3600 + scopesClaim: "scope" # enable scope enforcement (Keycloak) +``` + +### Discovery Endpoint + +The server exposes `GET /auth/oidc-config` which returns the configured OIDC issuer and audience. This allows CLI auto-discovery during `gateway add`. + +## Provider Examples + +### Keycloak + +```bash +openshell gateway start \ + --oidc-issuer http://keycloak:8180/realms/openshell +# Defaults work: realm_access.roles, openshell-admin, openshell-user +``` + +### Microsoft Entra ID + +Register an app in Azure Portal with app roles `OpenShell.Admin` and `OpenShell.User`. With Entra ID the client ID (the SPA/public app registration) and audience (the API app registration, e.g. `api://openshell`) are typically different: + +```bash +openshell gateway start \ + --oidc-issuer https://login.microsoftonline.com/{tenant-id}/v2.0 \ + --oidc-audience api://openshell \ + --oidc-client-id {client-id} \ + --oidc-roles-claim roles \ + --oidc-admin-role OpenShell.Admin \ + --oidc-user-role OpenShell.User +``` + +CLI registration (separate client ID and audience): + +```bash +openshell gateway add https://gateway:8080 \ + --oidc-issuer https://login.microsoftonline.com/{tenant-id}/v2.0 \ + --oidc-client-id {client-id} \ + --oidc-audience api://openshell +``` + +### Okta + +Create an authorization server with a `groups` claim, then: + +```bash +openshell gateway start \ + --oidc-issuer https://dev-xxxxx.okta.com/oauth2/default \ + --oidc-roles-claim groups \ + --oidc-admin-role openshell-admin \ + --oidc-user-role openshell-user +``` + +### GitHub (Authentication Only) + +GitHub's OIDC tokens (from Actions) don't carry roles. Use empty role names to skip RBAC — any valid GitHub JWT is authorized: + +```bash +openshell gateway start \ + --oidc-issuer https://token.actions.githubusercontent.com \ + --oidc-audience https://github.com/{org} \ + --oidc-admin-role "" \ + --oidc-user-role "" +``` + +## CLI Commands + +### Register an OIDC Gateway + +```bash +openshell gateway add http://gateway:8080 \ + --oidc-issuer http://keycloak:8180/realms/openshell + +# With custom client ID: +openshell gateway add http://gateway:8080 \ + --oidc-issuer http://keycloak:8180/realms/openshell \ + --oidc-client-id my-client + +# With separate client ID and audience (e.g. Entra ID): +openshell gateway add http://gateway:8080 \ + --oidc-issuer https://login.microsoftonline.com/{tenant-id}/v2.0 \ + --oidc-client-id {client-id} \ + --oidc-audience api://openshell +``` + +### Start a K3s Gateway with OIDC + +```bash +openshell gateway start \ + --oidc-issuer http://keycloak:8180/realms/openshell \ + --plaintext + +# With RBAC configuration: +openshell gateway start \ + --oidc-issuer http://keycloak:8180/realms/openshell \ + --oidc-client-id openshell-cli \ + --oidc-roles-claim realm_access.roles \ + --oidc-admin-role openshell-admin \ + --oidc-user-role openshell-user +``` + +### Authenticate + +```bash +# Interactive (opens browser) +openshell gateway login +# Expected: ✓ Authenticated to gateway 'openshell' as admin@test + +# CI / automation +OPENSHELL_OIDC_CLIENT_SECRET=secret openshell gateway login +``` + +### Logout + +```bash +openshell gateway logout +# Expected: ✓ Logged out of gateway 'openshell' +``` + +## Keycloak Setup + +### Realm Configuration + +The `scripts/keycloak-realm.json` file provides a pre-configured realm for development: + +- **Realm**: `openshell` +- **Clients**: + - `openshell-cli` — Public client, Authorization Code + PKCE, redirect URIs `http://127.0.0.1:*` + - `openshell-ci` — Confidential client, Client Credentials grant, secret `ci-test-secret` +- **Roles**: `openshell-admin`, `openshell-user` +- **Test Users**: + - `admin@test` / `admin` (roles: `openshell-admin`, `openshell-user`) + - `user@test` / `user` (roles: `openshell-user`) + +### Dev Server + +```bash +# Start Keycloak on port 8180 +./scripts/keycloak-dev.sh start + +# Check status +./scripts/keycloak-dev.sh status + +# Stop +./scripts/keycloak-dev.sh stop +``` + +Admin console: `http://localhost:8180/admin` (admin/admin). + +## Coexistence with Other Auth Modes + +OIDC is additive — it does not replace mTLS or Cloudflare Access. When OIDC is configured, the `AuthGrpcRouter` processes requests through the three-category classification: + +``` +Request arrives + | + +-- Strip x-openshell-auth-source (anti-spoofing) + | + +-- OIDC not configured? --> Pass through (mTLS/plaintext fallback) + | + +-- Unauthenticated method? --> Pass through + | + +-- Sandbox-secret method? + | +-- Valid x-sandbox-secret --> Mark auth-source, pass through + | +-- Invalid/missing --> UNAUTHENTICATED + | + +-- Dual-auth method? + | +-- Valid x-sandbox-secret --> Mark auth-source, pass through + | +-- No sandbox secret --> Fall through to Bearer + | + +-- Has "authorization: Bearer" header? + | +-- Validate JWT --> Check RBAC --> Check scopes (if enabled) --> Authenticated (OIDC) + | +-- Invalid JWT --> UNAUTHENTICATED + | + +-- No bearer header --> UNAUTHENTICATED +``` + +The CLI determines which auth mode to use based on `auth_mode` in gateway metadata. Only one mode is active per gateway registration. + +## Key Files + +| Component | File | +|---|---| +| Server OIDC validation + method classification | `crates/openshell-server/src/oidc.rs` | +| Server auth middleware | `crates/openshell-server/src/multiplex.rs` (`AuthGrpcRouter`) | +| Server authorization (RBAC) | `crates/openshell-server/src/authz.rs` (`AuthzPolicy`) | +| Sandbox-secret scope enforcement | `crates/openshell-server/src/grpc/policy.rs` (`validate_sandbox_secret_update`) | +| Server config | `crates/openshell-core/src/config.rs` (`OidcConfig`) | +| Server CLI flags | `crates/openshell-server/src/main.rs` | +| Server discovery endpoint | `crates/openshell-server/src/auth.rs` (`/auth/oidc-config`) | +| CLI OIDC flows | `crates/openshell-cli/src/oidc_auth.rs` | +| CLI interceptor | `crates/openshell-cli/src/tls.rs` (`EdgeAuthInterceptor`) | +| CLI auth dispatch | `crates/openshell-cli/src/main.rs` (`apply_auth`) | +| CLI gateway commands | `crates/openshell-cli/src/run.rs` (`gateway_add`, `gateway_login`) | +| Token storage | `crates/openshell-bootstrap/src/oidc_token.rs` | +| Gateway metadata | `crates/openshell-bootstrap/src/metadata.rs` | +| Bootstrap pipeline | `crates/openshell-bootstrap/src/lib.rs`, `docker.rs` | +| K3s entrypoint | `deploy/docker/cluster-entrypoint.sh` | +| HelmChart template | `deploy/kube/manifests/openshell-helmchart.yaml` | +| Helm values | `deploy/helm/openshell/values.yaml` | +| Helm statefulset | `deploy/helm/openshell/templates/statefulset.yaml` | +| Keycloak dev script | `scripts/keycloak-dev.sh` | +| Keycloak realm config | `scripts/keycloak-realm.json` | diff --git a/architecture/oidc-local-testing.md b/architecture/oidc-local-testing.md new file mode 100644 index 0000000000..160636a9e9 --- /dev/null +++ b/architecture/oidc-local-testing.md @@ -0,0 +1,575 @@ +# OIDC Local Testing Guide + +Step-by-step instructions for testing OIDC/Keycloak authentication locally, +including both standalone server testing and full end-to-end K3s testing. + +## Prerequisites + +- Docker or Podman +- Rust toolchain (edition 2024, rust 1.88+) +- `grpcurl` (for raw gRPC testing) +- `jq` (for JSON parsing) + +## 1. Start Keycloak + +```bash +mise run keycloak +``` + +Wait for "Keycloak is ready." The script prints connection info including test users. + +Verify: + +```bash +curl -s http://localhost:8180/realms/openshell/.well-known/openid-configuration | jq .issuer +# Expected: "http://localhost:8180/realms/openshell" +``` + +## 2. Standalone Server Testing (No K3s) + +Start the server directly with OIDC enabled. No Kubernetes cluster required. + +```bash +cargo run -p openshell-server -- \ + --disable-tls \ + --db-url sqlite:/tmp/openshell-test.db \ + --ssh-handshake-secret test \ + --oidc-issuer http://localhost:8180/realms/openshell +``` + +You should see: + +``` +OIDC JWT validation enabled (issuer: http://localhost:8180/realms/openshell) +Server listening address=0.0.0.0:8080 +``` + +K8s compute driver warnings are expected and non-fatal. + +### 2a. Test Health (unauthenticated — should succeed) + +```bash +grpcurl -plaintext -import-path proto -proto openshell.proto \ + 127.0.0.1:8080 openshell.v1.OpenShell/Health +# Expected: SERVICE_STATUS_HEALTHY +``` + +### 2b. Test without token (should fail) + +```bash +grpcurl -plaintext -import-path proto -proto openshell.proto \ + 127.0.0.1:8080 openshell.v1.OpenShell/ListSandboxes +# Expected: Code: Unauthenticated, Message: missing authorization header +``` + +### 2c. Get tokens from Keycloak + +```bash +ADMIN_TOKEN=$(curl -s -X POST http://localhost:8180/realms/openshell/protocol/openid-connect/token \ + -d 'grant_type=password&client_id=openshell-cli&username=admin@test&password=admin' \ + | jq -r .access_token) + +USER_TOKEN=$(curl -s -X POST http://localhost:8180/realms/openshell/protocol/openid-connect/token \ + -d 'grant_type=password&client_id=openshell-cli&username=user@test&password=user' \ + | jq -r .access_token) +``` + +### 2d. Test authenticated access + +```bash +# Admin can list sandboxes +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $ADMIN_TOKEN" \ + 127.0.0.1:8080 openshell.v1.OpenShell/ListSandboxes +# Expected: {} (empty list) + +# User can list sandboxes +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $USER_TOKEN" \ + 127.0.0.1:8080 openshell.v1.OpenShell/ListSandboxes +# Expected: {} (empty list) +``` + +### 2e. Test RBAC + +```bash +# User CANNOT create provider (requires openshell-admin) +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $USER_TOKEN" \ + -d '{"provider":{"name":"test","type":"claude","credentials":{"key":"val"}}}' \ + 127.0.0.1:8080 openshell.v1.OpenShell/CreateProvider +# Expected: Code: PermissionDenied, Message: role 'openshell-admin' required + +# Admin CAN create provider +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $ADMIN_TOKEN" \ + -d '{"provider":{"name":"test","type":"claude","credentials":{"key":"val"}}}' \ + 127.0.0.1:8080 openshell.v1.OpenShell/CreateProvider +# Expected: success +``` + +### 2f. Test sandbox secret auth + +```bash +# Correct secret — should succeed (returns an empty bundle when no routes are configured) +grpcurl -plaintext -import-path proto -proto inference.proto \ + -H "x-sandbox-secret: test" \ + 127.0.0.1:8080 openshell.inference.v1.Inference/GetInferenceBundle +# Expected: success with { "routes": [], ... } + +# Wrong secret — should fail at auth +grpcurl -plaintext -import-path proto -proto inference.proto \ + -H "x-sandbox-secret: wrong" \ + 127.0.0.1:8080 openshell.inference.v1.Inference/GetInferenceBundle +# Expected: Code: Unauthenticated, Message: invalid sandbox secret + +# No secret — should fail at auth +grpcurl -plaintext -import-path proto -proto inference.proto \ + 127.0.0.1:8080 openshell.inference.v1.Inference/GetInferenceBundle +# Expected: Code: Unauthenticated, Message: sandbox secret required +``` + +### 2g. Test OIDC discovery endpoint + +```bash +curl -s http://127.0.0.1:8080/auth/oidc-config | jq . +# Expected: {"audience":"openshell-cli","issuer":"http://localhost:8180/realms/openshell"} +``` + +Stop the standalone server (Ctrl+C) before proceeding to K3s testing. + +## 3. CLI OIDC Flow (Standalone) + +With the standalone server running from step 2: + +```bash +# Register the gateway with OIDC auth +cargo run -p openshell-cli --features bundled-z3 -- gateway add http://127.0.0.1:8080 \ + --oidc-issuer http://localhost:8180/realms/openshell + +# Browser opens to Keycloak. Login with: admin@test / admin +# Expected: ✓ Authenticated to gateway 'localhost' as admin@test + +# Verify stored token +cat ~/.config/openshell/gateways/127.0.0.1/oidc_token.json | jq . + +# Test authenticated CLI command +cargo run -p openshell-cli --features bundled-z3 -- sandbox list +``` + +### Test client credentials (CI mode) + +The CI client (`openshell-ci`) is separate from the interactive client (`openshell-cli`). +Register the gateway with the CI client ID first: + +```bash +cargo run -p openshell-cli --features bundled-z3 -- gateway add http://127.0.0.1:8080 \ + --oidc-issuer http://localhost:8180/realms/openshell \ + --oidc-client-id openshell-ci + +OPENSHELL_OIDC_CLIENT_SECRET=ci-test-secret \ +cargo run -p openshell-cli --features bundled-z3 -- gateway login +# Expected: ✓ Authenticated to gateway (no browser opened) +``` + +### Test logout + +```bash +cargo run -p openshell-cli --features bundled-z3 -- gateway logout +# Expected: ✓ Logged out of gateway + +cargo run -p openshell-cli --features bundled-z3 -- sandbox list +# Expected: error (no token) +``` + +## 4. End-to-End K3s Testing + +This deploys a full K3s cluster with OIDC enforcement and tests sandbox +creation, RBAC, login/logout, and token expiry. + +### 4a. Bootstrap the cluster with OIDC + +Keycloak runs on the host. The K3s container reaches it via the host IP. +The `OPENSHELL_OIDC_ISSUER` env var tells the deploy script to pass the +issuer to the Helm chart so the gateway starts with JWT validation enabled. + +```bash +HOST_IP=$(hostname -I | awk '{print $1}') +OPENSHELL_OIDC_ISSUER="http://${HOST_IP}:8180/realms/openshell" \ +OPENSHELL_OIDC_SCOPES="openshell:all" \ +mise run cluster +``` + +Add `OPENSHELL_OIDC_SCOPES_CLAIM="scope"` to also enable scope enforcement. +The `OPENSHELL_OIDC_SCOPES` value is stored in gateway metadata so `gateway login` +requests these scopes automatically. + +Wait for "Deploy complete!" and verify OIDC is active: + +```bash +CONTAINER=$(docker ps --format '{{.Names}}' | grep openshell-cluster) +docker exec $CONTAINER kubectl -n openshell logs openshell-0 | grep OIDC +# Expected: OIDC JWT validation enabled (issuer: http://...) +``` + +### 4b. Login to the gateway + +The bootstrap step above configures the gateway metadata with the OIDC +issuer automatically. Authenticate with Keycloak: + +```bash +openshell gateway login +# Login with: admin@test / admin +# Expected: ✓ Authenticated to gateway 'openshell' as admin@test +``` + +### 4c. Create and list sandboxes + +```bash +# Login as admin +openshell gateway login +# Login with: admin@test / admin +# Expected: ✓ Authenticated to gateway 'openshell' as admin@test + +# Create a sandbox +openshell sandbox create +# Expected: Created sandbox: + +# List sandboxes +openshell sandbox list +# Expected: shows the created sandbox +``` + +### 4d. Verify authentication enforcement + +```bash +# Logout +openshell gateway logout +# Expected: ✓ Logged out of gateway 'openshell' + +# Should fail without token +openshell sandbox list +# Expected: Unauthenticated error + +# Login again +openshell gateway login +# Login with: admin@test / admin + +# Should work again +openshell sandbox list +# Expected: shows sandboxes +``` + +### 4e. Verify token expiry + +Keycloak access tokens expire after 5 minutes by default. + +```bash +# Wait 5+ minutes, then: +openshell sandbox list +# Expected: Unauthenticated: ExpiredSignature + +# Re-login +openshell gateway login +openshell sandbox list +# Expected: success +``` + +### 4f. Verify RBAC + +```bash +# Login as admin +openshell gateway login +# Login with: admin@test / admin + +# Admin can create a provider +openshell provider create \ + --name test-provider --type claude --credential API_KEY=test123 +# Expected: success + +# Login as user (openshell-user only, no openshell-admin) +openshell gateway login +# Login with: user@test / user +# Expected: ✓ Authenticated to gateway 'openshell' as user@test + +# User can list sandboxes +openshell sandbox list +# Expected: success + +# User can list providers +openshell provider list +# Expected: shows test-provider + +# User CANNOT create a provider +openshell provider create \ + --name blocked --type claude --credential API_KEY=nope +# Expected: PermissionDenied: role 'openshell-admin' required + +# User CANNOT delete a provider +openshell provider delete test-provider +# Expected: PermissionDenied: role 'openshell-admin' required + +# User CAN create sandboxes +openshell sandbox create +# Expected: success +``` + +### 4g. Test client credentials (CI mode) + +The CI client uses `openshell-ci` (confidential) instead of `openshell-cli` (public). +Update the gateway metadata to use the CI client, then login: + +```bash +jq '.oidc_client_id = "openshell-ci"' \ + ~/.config/openshell/gateways/openshell/metadata.json > /tmp/meta.json \ + && mv /tmp/meta.json ~/.config/openshell/gateways/openshell/metadata.json + +OPENSHELL_OIDC_CLIENT_SECRET=ci-test-secret \ +openshell gateway login +# Expected: ✓ Authenticated to gateway 'openshell' (no browser) + +openshell sandbox list +# Expected: success + +# Restore interactive client for further testing +jq '.oidc_client_id = "openshell-cli"' \ + ~/.config/openshell/gateways/openshell/metadata.json > /tmp/meta.json \ + && mv /tmp/meta.json ~/.config/openshell/gateways/openshell/metadata.json +``` + +### 4h. Clean up sandboxes + +```bash +# Login as admin to clean up +openshell gateway login +# Login with: admin@test / admin + +openshell sandbox list +# Note sandbox names, then: +openshell sandbox delete + +openshell provider delete test-provider +``` + +## 5. Scope-Based Permissions Testing + +Scopes provide fine-grained, per-method access control on top of roles. This section tests scope enforcement using both the standalone server and K3s. + +### 5a. Standalone server with scope enforcement + +```bash +cargo run -p openshell-server -- \ + --disable-tls \ + --db-url sqlite:/tmp/openshell-scopes-test.db \ + --ssh-handshake-secret test \ + --oidc-issuer http://localhost:8180/realms/openshell \ + --oidc-scopes-claim scope +``` + +### 5b. Get tokens with specific scopes + +```bash +# Token with sandbox scopes only +TOKEN_SANDBOX=$(curl -s -X POST http://localhost:8180/realms/openshell/protocol/openid-connect/token \ + -d 'grant_type=password&client_id=openshell-cli&username=admin@test&password=admin' \ + -d 'scope=openid sandbox:read sandbox:write' \ + | jq -r .access_token) + +# Token with all scopes +TOKEN_ALL=$(curl -s -X POST http://localhost:8180/realms/openshell/protocol/openid-connect/token \ + -d 'grant_type=password&client_id=openshell-cli&username=admin@test&password=admin' \ + -d 'scope=openid openshell:all' \ + | jq -r .access_token) + +# Token without OpenShell scopes (roles-only) +TOKEN_NO_SCOPES=$(curl -s -X POST http://localhost:8180/realms/openshell/protocol/openid-connect/token \ + -d 'grant_type=password&client_id=openshell-cli&username=admin@test&password=admin' \ + | jq -r .access_token) +``` + +### 5c. Inspect tokens + +```bash +# Verify scopes are in the JWT +echo "$TOKEN_SANDBOX" | cut -d. -f2 | base64 -d 2>/dev/null | jq '{scope, realm_access, preferred_username}' +# Expected: scope contains "sandbox:read sandbox:write", realm_access has roles, preferred_username is set + +echo "$TOKEN_NO_SCOPES" | cut -d. -f2 | base64 -d 2>/dev/null | jq '.scope' +# Expected: "openid email profile" (no OpenShell scopes) +``` + +### 5d. Test scope enforcement with grpcurl + +```bash +# Sandbox-scoped token — ListSandboxes should work +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $TOKEN_SANDBOX" \ + 127.0.0.1:8080 openshell.v1.OpenShell/ListSandboxes +# Expected: success (empty list) + +# Sandbox-scoped token — ListProviders should FAIL +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $TOKEN_SANDBOX" \ + 127.0.0.1:8080 openshell.v1.OpenShell/ListProviders +# Expected: PermissionDenied: scope 'provider:read' required + +# openshell:all token — everything works +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $TOKEN_ALL" \ + 127.0.0.1:8080 openshell.v1.OpenShell/ListProviders +# Expected: success + +# No-scopes token — denied +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $TOKEN_NO_SCOPES" \ + 127.0.0.1:8080 openshell.v1.OpenShell/ListSandboxes +# Expected: PermissionDenied: scope 'sandbox:read' required +``` + +### 5e. Test CLI with scopes + +Stop the standalone server. Register a gateway with scopes: + +```bash +openshell gateway add http://127.0.0.1:8080 \ + --oidc-issuer http://localhost:8180/realms/openshell \ + --oidc-scopes "sandbox:read sandbox:write" +``` + +Or for K3s testing, pass `OPENSHELL_OIDC_SCOPES` during bootstrap: + +```bash +HOST_IP=$(hostname -I | awk '{print $1}') +OPENSHELL_OIDC_ISSUER="http://${HOST_IP}:8180/realms/openshell" \ +OPENSHELL_OIDC_SCOPES_CLAIM="scope" \ +OPENSHELL_OIDC_SCOPES="sandbox:read sandbox:write" \ +mise run cluster +``` + +Then login and test: + +```bash +openshell gateway login +# Login with: admin@test / admin + +openshell sandbox list # should work (has sandbox:read) +openshell provider list # should fail (no provider:read scope) +``` + +### 5f. Test openshell:all via CLI + +For K3s, restart the cluster with `openshell:all`: + +```bash +mise run cluster:stop +HOST_IP=$(hostname -I | awk '{print $1}') +OPENSHELL_OIDC_ISSUER="http://${HOST_IP}:8180/realms/openshell" \ +OPENSHELL_OIDC_SCOPES_CLAIM="scope" \ +OPENSHELL_OIDC_SCOPES="openshell:all" \ +mise run cluster + +openshell gateway login +openshell sandbox list # should work +openshell provider list # should work +``` + +### 5g. Test CI client credentials with scopes + +```bash +OPENSHELL_OIDC_CLIENT_SECRET=ci-test-secret openshell gateway login +# openshell-ci has openshell:all as a default scope + +openshell sandbox list # should work +openshell provider list # should work +``` + +### 5h. Test without scope enforcement (default behavior preserved) + +Restart the server WITHOUT `--oidc-scopes-claim`: + +```bash +cargo run -p openshell-server -- \ + --disable-tls \ + --db-url sqlite:/tmp/openshell-noscopes-test.db \ + --ssh-handshake-secret test \ + --oidc-issuer http://localhost:8180/realms/openshell +``` + +```bash +# Token without scopes should work (roles-only mode) +grpcurl -plaintext -import-path proto -proto openshell.proto \ + -H "authorization: Bearer $TOKEN_NO_SCOPES" \ + 127.0.0.1:8080 openshell.v1.OpenShell/ListSandboxes +# Expected: success — scopes are not enforced +``` + +## 6. Cleanup + +```bash +# Stop the cluster +mise run cluster:stop + +# Stop Keycloak +mise run keycloak:stop +``` + +## Test Users + +| Username | Password | Roles | +|---|---|---| +| `admin@test` | `admin` | `openshell-admin`, `openshell-user` | +| `user@test` | `user` | `openshell-user` | + +## OIDC Clients + +| Client ID | Type | Grant | Secret | +|---|---|---|---| +| `openshell-cli` | Public | Auth Code + PKCE | N/A | +| `openshell-ci` | Confidential | Client Credentials | `ci-test-secret` | + +## Method Authentication Categories + +| Category | Methods | Auth Mechanism | +|---|---|---| +| Unauthenticated | Health, gRPC reflection | None | +| Sandbox-secret | GetSandboxConfig, GetSandboxProviderEnvironment, ReportPolicyStatus, PushSandboxLogs, SubmitPolicyAnalysis | `x-sandbox-secret` header | +| Dual-auth | UpdateConfig | Bearer token OR `x-sandbox-secret` | +| OIDC Bearer | All other RPCs | `authorization: Bearer ` | + +## Role Requirements + +| Operation | Required Role | +|---|---| +| Sandbox create, list, delete, exec, SSH | `openshell-user` | +| Provider list, get | `openshell-user` | +| Provider create, update, delete | `openshell-admin` | +| Global config/policy updates | `openshell-admin` | +| Draft policy approvals | `openshell-admin` | + +## Troubleshooting + +**"missing authorization header"** — No OIDC token stored. Run `openshell gateway login`. + +**"invalid token: ExpiredSignature"** — Token expired (default 5 min). Run `openshell gateway login`. + +**"PermissionDenied: role 'openshell-admin' required"** — Logged in as a user without the admin role. Login as `admin@test`. + +**"sandbox secret required for this method"** — A sandbox-to-server RPC was called without the `x-sandbox-secret` header. + +**"OIDC discovery request failed"** — Server can't reach Keycloak. Use the host IP (not `localhost`) for K3s deployments. + +**"invalid token: unknown signing key"** — JWKS key mismatch. Restart the server to refresh the cache. + +**No "OIDC JWT validation enabled" in K3s logs** — The `OPENSHELL_OIDC_ISSUER` env var was not set when deploying. Re-run `OPENSHELL_OIDC_ISSUER="http://:8180/realms/openshell" mise run cluster gateway` to rebuild and redeploy with OIDC enabled. + +**"InvalidIssuer"** — The issuer URL in the OIDC token does not match the server's configured issuer. Ensure the gateway metadata `oidc_issuer` uses the same URL the server was started with (typically the host IP, not `localhost`). + +**"connection refused" with grpcurl** — On Fedora/systems where `localhost` resolves to IPv6, use `127.0.0.1` instead of `localhost`. + +**"no such table: objects"** — Using `sqlite::memory:` which doesn't run migrations. Use a file path like `sqlite:/tmp/openshell-test.db`. + +**"scope 'X' required"** — The server has `--oidc-scopes-claim` enabled and the token is missing the required scope. Either request the scope during login (`--oidc-scopes "sandbox:read sandbox:write"`) or use `openshell:all` for full access. + +**Token has scopes but server doesn't enforce them** — The server was started without `--oidc-scopes-claim`. Add `--oidc-scopes-claim scope` (for Keycloak) to enable enforcement. + +**Scopes missing from token after Keycloak login** — The browser may have reused an old Keycloak session with the previous scope set. Sign out at `http://localhost:8180/realms/openshell/account/#/` and re-run `openshell gateway login`. diff --git a/architecture/podman-driver.md b/architecture/podman-driver.md new file mode 100644 index 0000000000..c6fcfdb0db --- /dev/null +++ b/architecture/podman-driver.md @@ -0,0 +1,271 @@ +# Podman Compute Driver + +The Podman compute driver manages sandbox containers via the Podman REST API over a Unix socket. It targets single-machine and developer environments where rootless container isolation is preferred over a full Kubernetes cluster. The driver runs in-process within the gateway server and delegates all sandbox isolation enforcement to the `openshell-sandbox` supervisor binary, which is sideloaded into each container via an OCI image volume mount. + +## Source File Index + +All paths are relative to `crates/openshell-driver-podman/src/`. + +| File | Purpose | +|------|---------| +| `lib.rs` | Crate root; declares modules and re-exports `PodmanComputeConfig`, `PodmanComputeDriver`, `ComputeDriverService` | +| `main.rs` | Standalone binary entrypoint; parses CLI args/env vars, constructs the driver, starts a gRPC server with graceful shutdown | +| `driver.rs` | Core `PodmanComputeDriver` -- sandbox lifecycle (create/stop/delete/list/get), endpoint resolution, GPU detection, rootless pre-flight checks | +| `client.rs` | `PodmanClient` -- async HTTP/1.1 client over Unix socket for the Podman libpod REST API (containers, volumes, networks, secrets, images, events, system info) | +| `container.rs` | Container spec construction -- labels, env vars, resource limits, capabilities, seccomp config, health checks, port mappings, image volumes, secret injection | +| `config.rs` | `PodmanComputeConfig` struct, `ImagePullPolicy` enum, default socket path resolution, `Debug` impl that redacts secrets | +| `grpc.rs` | `ComputeDriverService` -- tonic gRPC service mapping RPCs to driver methods, with error-to-Status conversion | +| `watcher.rs` | Watch stream -- initial state sync via container list, then live Podman events mapped to `WatchSandboxesEvent` protobuf messages | + +## Architecture + +The Podman driver is one of three `ComputeDriver` implementations. It communicates with the Podman daemon over a Unix socket and delegates sandbox isolation to the supervisor binary running inside each container. + +```mermaid +graph TB + CLI["openshell CLI"] -->|gRPC| GW["Gateway Server
(openshell-server)"] + GW -->|in-process| PD["PodmanComputeDriver"] + PD -->|HTTP/1.1
Unix socket| PA["Podman API"] + PA -->|OCI runtime
crun/runc| C["Sandbox Container"] + C -->|image volume
read-only| SV["Supervisor Binary
/opt/openshell/bin/openshell-sandbox"] + SV -->|creates| NS["Nested Network Namespace
veth pair + proxy"] + SV -->|enforces| LL["Landlock + seccomp"] + SV -->|gRPC callback| GW +``` + +### Driver Comparison + +| Aspect | Kubernetes | VM (libkrun) | Podman | +|--------|-----------|--------------|--------| +| Execution model | In-process | Standalone subprocess (gRPC over UDS) | In-process | +| Backend | K8s API (CRD + controller) | libkrun hypervisor (KVM/HVF) | Podman REST API (Unix socket) | +| Isolation boundary | Container (supervisor inside pod) | Hardware VM | Container (supervisor inside container) | +| Supervisor delivery | hostPath volume (read-only) | Embedded in rootfs tarball | OCI image volume (read-only) | +| Network model | Supervisor creates netns inside pod | gvproxy virtio-net (192.168.127.0/24) | Supervisor creates netns inside container | +| Credential injection | Plaintext env var + K8s Secret volume (0400) | Rootfs file copy (0600) + env vars | Podman `secret_env` API + env vars | +| GPU support | Yes (nvidia.com/gpu resource) | No | Yes (CDI device) | +| `stop_sandbox` | Unimplemented | Unimplemented | Implemented (graceful stop) | +| State storage | Kubernetes API (CRD) | In-memory HashMap + filesystem | Podman daemon (container state) | +| Endpoint resolution | Pod IP / cluster DNS | 127.0.0.1 + allocated port | 127.0.0.1 + ephemeral port | + +## Isolation Model + +The Podman driver provides the same four protection layers as the Kubernetes driver. The driver itself does not implement isolation primitives directly -- it configures the container so that the `openshell-sandbox` supervisor binary can enforce them at runtime. + +### Container Security Configuration + +The container spec (`container.rs`) sets: + +| Setting | Value | Rationale | +|---------|-------|-----------| +| `user` | `0:0` | Supervisor needs root for namespace creation, proxy setup, Landlock/seccomp | +| `cap_drop` | `ALL` | Drop all capabilities, then selectively add back | +| `cap_add` | `SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYSLOG, SETUID, SETGID, DAC_READ_SEARCH` | See capability breakdown below | +| `no_new_privileges` | `true` | Prevent privilege escalation after exec | +| `seccomp_profile_path` | `unconfined` | Supervisor installs its own policy-aware BPF filter; container-level profile would block Landlock/seccomp syscalls during setup | + +### Capability Breakdown + +The Kubernetes driver uses 4 capabilities (`SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYSLOG`). The Podman driver adds 3 more, all required for rootless operation: + +| Capability | Shared with K8s? | Purpose | +|------------|------------------|---------| +| `SYS_ADMIN` | Yes | seccomp filter installation, namespace creation, Landlock | +| `NET_ADMIN` | Yes | Network namespace veth setup, IP/route configuration | +| `SYS_PTRACE` | Yes | Reading `/proc//exe` and ancestor walk for binary identity | +| `SYSLOG` | Yes | Reading `/dev/kmsg` for bypass-detection diagnostics | +| `SETUID` | Podman only | `drop_privileges()` calls `setuid()` to sandbox user. Rootless `cap_drop: ALL` removes this from the bounding set. | +| `SETGID` | Podman only | `drop_privileges()` calls `setgid()` + `initgroups()`. Same rootless reason as SETUID. | +| `DAC_READ_SEARCH` | Podman only | Proxy reads `/proc//fd/` across UIDs for binary identity. In rootless Podman, supervisor (UID 0 in user namespace) and sandbox processes have different UIDs. | + +In the Kubernetes driver, these three capabilities are implicitly available because the kubelet does not drop them from the bounding set. In rootless Podman, `cap_drop: ALL` removes everything, requiring explicit re-addition. + +All capabilities are only available to the supervisor process. Sandbox child processes lose them after `setuid()` to the sandbox user in the `pre_exec` hook. + +## Supervisor Sideloading + +The supervisor binary is delivered to sandbox containers via Podman's OCI image volume mechanism, distinct from both the Kubernetes hostPath approach and the VM's embedded rootfs. + +```mermaid +sequenceDiagram + participant D as PodmanComputeDriver + participant P as Podman API + participant C as Sandbox Container + + D->>P: pull_image("openshell/supervisor:latest", "missing") + D->>P: create_container(spec with image_volumes) + Note over P: Podman resolves image_volumes at
libpod layer before OCI spec generation + P->>C: Mount supervisor image at /opt/openshell/bin (read-only) + D->>P: start_container + C->>C: entrypoint: /opt/openshell/bin/openshell-sandbox +``` + +The supervisor image is a `FROM scratch` image containing only the prebuilt `openshell-sandbox` binary. It is built by the `supervisor` target in `deploy/docker/Dockerfile.images`. The `image_volumes` field in the container spec mounts this image's filesystem at `/opt/openshell/bin` with `rw: false`, making it a read-only overlay that the sandbox cannot tamper with. + +## TLS + +When the Podman driver's TLS configuration is set (`tls_ca`, `tls_cert`, `tls_key` in `PodmanComputeConfig`), the driver: + +1. Switches the auto-detected endpoint scheme from `http://` to `https://` +2. Bind-mounts the client cert files (read-only) into the container at `/etc/openshell/tls/client/` +3. Sets `OPENSHELL_TLS_CA`, `OPENSHELL_TLS_CERT`, `OPENSHELL_TLS_KEY` env vars pointing to the container-side paths + +The supervisor reads these env vars and uses them to establish an mTLS connection back to the gateway. + +The RPM packaging auto-generates a self-signed PKI on first start via `init-pki.sh`. Client certs are placed in the CLI auto-discovery directory (`~/.config/openshell/gateways/openshell/mtls/`) so the CLI connects with mTLS without manual configuration. See `deploy/rpm/CONFIGURATION.md` for the full RPM configuration reference and `deploy/rpm/QUICKSTART.md` for the quick start guide. + +## Network Model + +Sandbox network isolation uses a two-layer approach: a Podman bridge network for container-to-host communication, and a nested network namespace (created by the supervisor) for sandbox process isolation. + +```mermaid +graph TB + subgraph Host + GW["Gateway Server
127.0.0.1:8080"] + PS["Podman Socket"] + end + + subgraph Bridge["Podman Bridge Network (10.89.x.x)"] + subgraph Container["Sandbox Container"] + SV["Supervisor
(root in user ns)"] + subgraph NestedNS["Nested Network Namespace"] + SP["Sandbox Process
(sandbox user)"] + VE2["veth1: 10.200.0.2"] + end + VE1["veth0: 10.200.0.1
(CONNECT proxy)"] + SV --- VE1 + VE1 ---|veth pair| VE2 + end + end + + GW -.->|SSH via supervisor relay
gRPC session| SV + SV -->|gRPC callback via
host.containers.internal| GW + SP -->|all egress via proxy| VE1 +``` + +Key points: + +- **Bridge network**: Created by `client.ensure_network()` with DNS enabled. Containers on the bridge can see each other at L3, but sandbox processes cannot because they are isolated inside the nested netns. +- **Nested netns**: The supervisor creates a private `NetworkNamespace` with a veth pair (10.200.0.1/24 <-> 10.200.0.2/24). Sandbox processes enter this netns via `setns(fd, CLONE_NEWNET)` in the `pre_exec` hook, forcing all traffic through the CONNECT proxy. +- **Port publishing**: SSH uses `host_port: 0` (ephemeral port assignment) for health checks and debug access. The gateway SSH tunnel uses the supervisor relay (`supervisor_sessions.open_relay()`) rather than connecting directly to the published port. +- **Host gateway**: `host.containers.internal:host-gateway` in `/etc/hosts` allows containers to reach the gateway server on the host. +- **nsenter**: The supervisor uses `nsenter --net=` instead of `ip netns exec` for namespace operations, avoiding the sysfs remount that fails in rootless containers. + +## Supervisor relay (SSH Unix socket) + +Podman now follows the same end-to-end contract as the Kubernetes and VM drivers for the in-container SSH relay: **gateway config → `PodmanComputeConfig` → sandbox environment → supervisor session registration on that path**. + +1. `openshell-core` `Config::sandbox_ssh_socket_path` (gateway YAML / defaults) is copied into `PodmanComputeConfig::sandbox_ssh_socket_path` when the gateway builds the in-process driver (`crates/openshell-server/src/lib.rs`, `ComputeDriverKind::Podman`). +2. `build_env()` in `container.rs` sets `OPENSHELL_SSH_SOCKET_PATH` to that value, alongside required vars `OPENSHELL_ENDPOINT` and `OPENSHELL_SANDBOX_ID` (and `OPENSHELL_SANDBOX`, etc.). These driver-controlled entries overwrite user template env to prevent spoofing. +3. The supervisor reads `OPENSHELL_SSH_SOCKET_PATH` and uses it for the Unix socket the gateway’s SSH stack bridges to. The standalone `openshell-driver-podman` binary sets the same struct field from `OPENSHELL_SANDBOX_SSH_SOCKET_PATH` (`main.rs`). + +## Credential Injection + +The SSH handshake secret is injected via Podman's `secret_env` API rather than a plaintext environment variable. + +| Credential | Mechanism | Visible in `inspect`? | Visible in `/proc//environ`? | +|------------|-----------|----------------------|----------------------------------| +| SSH handshake secret | Podman `secret_env` (created via secrets API, referenced by name) | No | Yes (supervisor only; scrubbed from children) | +| Sandbox identity (`OPENSHELL_SANDBOX_ID`, etc.) | Plaintext env var | Yes | Yes | +| gRPC endpoint (`OPENSHELL_ENDPOINT`) | Plaintext env var, override-protected | Yes | Yes | +| Supervisor relay socket path (`OPENSHELL_SSH_SOCKET_PATH`) | Plaintext env var, override-protected (same value as `PodmanComputeConfig::sandbox_ssh_socket_path`) | Yes | Yes | + +The `build_env()` function in `container.rs` inserts user-supplied variables first, then unconditionally overwrites all security-critical variables to prevent spoofing via sandbox templates: `OPENSHELL_SANDBOX`, `OPENSHELL_SANDBOX_ID`, `OPENSHELL_ENDPOINT`, `OPENSHELL_SSH_SOCKET_PATH`, `OPENSHELL_SSH_HANDSHAKE_SKEW_SECS`, `OPENSHELL_CONTAINER_IMAGE`, `OPENSHELL_SANDBOX_COMMAND`. + +The `PodmanComputeConfig::Debug` impl redacts the handshake secret as `[REDACTED]`. + +## Sandbox Lifecycle + +### Creation Flow + +```mermaid +sequenceDiagram + participant GW as Gateway + participant D as PodmanComputeDriver + participant P as Podman API + + GW->>D: create_sandbox(DriverSandbox) + D->>D: validate name + id + D->>D: validated_container_name() + + D->>P: pull_image(supervisor, "missing") + D->>P: pull_image(sandbox_image, policy) + + D->>P: create_secret(handshake) + Note over D: On failure below, rollback secret + + D->>P: create_volume(workspace) + Note over D: On failure below, rollback volume + secret + + D->>P: create_container(spec) + alt Conflict (409) + D->>P: remove_volume + remove_secret + D-->>GW: AlreadyExists + end + Note over D: On failure below, rollback container + volume + secret + + D->>P: start_container + D-->>GW: Ok +``` + +Each step rolls back all previously-created resources on failure. The Conflict path (409 from container creation) cleans up the volume and secret because they are keyed by the new sandbox's ID, not the conflicting container's. + +### Readiness and health + +The container `healthconfig` in `container.rs` marks the sandbox healthy when **any** of these signals succeeds: legacy file marker `/var/run/openshell-ssh-ready`, **or** `test -S` on the configured supervisor Unix socket path (`sandbox_ssh_socket_path` / `OPENSHELL_SSH_SOCKET_PATH`), **or** the prior TCP check (`ss` listening on the in-container SSH port). That allows relay-only readiness when the supervisor exposes the socket without the old marker or published-port signal. + +### Deletion Flow + +1. Validate `sandbox_name` and stable `sandbox_id` from `DeleteSandboxRequest` +2. Best-effort inspect cross-checks the container label when present, but cleanup remains keyed by the request `sandbox_id` +3. Best-effort stop (result ignored) +4. Force-remove container (`?force=true&v=true`) +5. Remove workspace volume derived from the request `sandbox_id` (warn on failure, continue) +6. Remove handshake secret derived from the request `sandbox_id` (warn on failure, continue) + +If the container is already gone during inspect or remove, the driver still performs idempotent volume/secret cleanup using the request `sandbox_id` and returns `Ok(false)` for the container-delete result. This prevents leaked Podman resources after out-of-band container removal or label drift. + +## Configuration + +| Environment Variable | CLI Flag | Default | Description | +|---------------------|----------|---------|-------------| +| `OPENSHELL_PODMAN_SOCKET` | `--podman-socket` | `$XDG_RUNTIME_DIR/podman/podman.sock` (Linux), `$HOME/.local/share/containers/podman/machine/podman.sock` (macOS) | Podman API Unix socket path | +| `OPENSHELL_SANDBOX_IMAGE` | `--sandbox-image` | (from gateway config) | Default OCI image for sandboxes | +| `OPENSHELL_SANDBOX_IMAGE_PULL_POLICY` | `--sandbox-image-pull-policy` | `missing` | Pull policy: `always`, `missing`, `never`, `newer` | +| `OPENSHELL_GRPC_ENDPOINT` | `--grpc-endpoint` | Auto-detected via `host.containers.internal` | Gateway gRPC endpoint for sandbox callbacks | +| `OPENSHELL_NETWORK_NAME` | `--network-name` | `openshell` | Podman bridge network name | +| `OPENSHELL_SANDBOX_SSH_PORT` | `--sandbox-ssh-port` | `2222` | SSH port inside the container | +| `OPENSHELL_SSH_HANDSHAKE_SECRET` | `--ssh-handshake-secret` | (required) | Shared secret for NSSH1 handshake | +| `OPENSHELL_SANDBOX_SSH_SOCKET_PATH` | `--sandbox-ssh-socket-path` | `/run/openshell/ssh.sock` | Standalone driver only: supervisor Unix socket path in `PodmanComputeConfig` (in-gateway Podman uses server `config.sandbox_ssh_socket_path`) | +| `OPENSHELL_STOP_TIMEOUT` | `--stop-timeout` | `10` | Container stop timeout in seconds (SIGTERM -> SIGKILL) | +| `OPENSHELL_SUPERVISOR_IMAGE` | `--supervisor-image` | `openshell/supervisor:latest` (struct default; standalone binary requires explicit value) | OCI image containing the supervisor binary | + +## Rootless-Specific Adaptations + +The Podman driver is designed for rootless operation. The following adaptations were made compared to the Kubernetes driver: + +1. **subuid/subgid pre-flight check**: `check_subuid_range()` in `driver.rs` warns operators if `/etc/subuid` or `/etc/subgid` entries are missing for the current user. Not a hard error because some systems use LDAP or other mechanisms. + +2. **cgroups v2 requirement**: The driver refuses to start if cgroups v1 is detected. Rootless Podman requires the unified cgroup hierarchy. + +3. **nsenter for namespace operations**: `run_ip_netns()` and `run_iptables_netns()` in `crates/openshell-sandbox/src/sandbox/linux/netns.rs` use `nsenter --net=` instead of `ip netns exec` to avoid the sysfs remount that requires real `CAP_SYS_ADMIN` in the host user namespace. + +4. **DAC_READ_SEARCH capability**: Required for the proxy to read `/proc//fd/` across UIDs within the user namespace. + +5. **SETUID/SETGID capabilities**: Required for `drop_privileges()` to call `setuid()`/`setgid()` after `cap_drop: ALL` removes them from the bounding set. + +6. **host.containers.internal**: Used instead of Docker's `host.docker.internal` for container-to-host communication. Injected via `hostadd` with Podman's `host-gateway` magic value. + +7. **Ephemeral port publishing**: SSH port uses `host_port: 0` because the bridge network IP (10.89.x.x) is not routable from the host in rootless mode. The published port is used for health checks and debug access; the gateway SSH tunnel uses the supervisor relay. + +8. **tmpfs at `/run/netns`**: A private tmpfs is mounted so the supervisor can create named network namespaces via `ip netns add`, which requires `/run/netns` to exist and be writable. + +## Implementation References + +- Gateway integration: `crates/openshell-server/src/compute/mod.rs` (`new_podman` and `PodmanComputeDriver` wiring) +- Server configuration: `crates/openshell-server/src/lib.rs` (`ComputeDriverKind::Podman` — builds `PodmanComputeConfig` including `sandbox_ssh_socket_path` from gateway `Config`) +- Gateway relay path: `openshell-core` `Config::sandbox_ssh_socket_path` in `crates/openshell-core/src/config.rs` +- SSRF mitigation: `crates/openshell-core/src/net.rs` (IP classification: `is_always_blocked_ip`, `is_internal_ip`), `crates/openshell-sandbox/src/proxy.rs` (runtime enforcement on CONNECT/forward proxy), `crates/openshell-server/src/grpc/policy.rs` (load-time validation via `validate_rule_not_always_blocked`) +- Sandbox supervisor: `crates/openshell-sandbox/src/` (Landlock, seccomp, netns, proxy -- shared by all drivers) +- Container engine abstraction: `tasks/scripts/container-engine.sh` (build/deploy support for Docker and Podman) +- Supervisor image build: `deploy/docker/Dockerfile.images` (`supervisor-output` target) diff --git a/architecture/podman-rootless-networking.md b/architecture/podman-rootless-networking.md new file mode 100644 index 0000000000..99d1626694 --- /dev/null +++ b/architecture/podman-rootless-networking.md @@ -0,0 +1,387 @@ +# Rootless Podman Networking + +Deep-dive into how networking works in the Podman compute driver when running rootless with pasta as the network backend. Covers the external tooling (Podman, Netavark, pasta, aardvark-dns), the three nested namespace layers, and the complete data paths for SSH, outbound traffic, and supervisor-to-gateway communication. + +For the general Podman driver architecture (lifecycle, API surface, driver comparison), see [podman-driver.md](podman-driver.md). + +## Component Stack + +Podman's networking is composed of four independent projects: + +| Component | Language | Role | +|-----------|----------|------| +| **Podman** | Go | Container runtime; orchestrates network lifecycle | +| **Netavark** | Rust | Network backend; creates interfaces, bridges, firewall rules | +| **aardvark-dns** | Rust | Authoritative DNS server for container name resolution (A/AAAA records) | +| **pasta** (part of passt) | C | User-mode networking; L2-to-L4 socket translation for rootless containers | + +The key split: rootful containers default to Netavark (bridge networking with real kernel interfaces), while rootless containers default to pasta (user-mode networking, no privileges needed). + +## How Netavark Works (Rootful) + +Netavark is invoked by Podman as an external binary. It reads a JSON network configuration from STDIN and executes one of three commands: + +- `netavark setup ` -- creates interfaces, assigns IPs, sets up firewall rules for NAT/port-forwarding +- `netavark teardown ` -- reverses setup; removes interfaces and firewall rules +- `netavark create` -- takes a partial network config and completes it (assigns subnets, gateways) + +For rootful bridge networking: + +1. Podman creates a network namespace for the container +2. Podman invokes `netavark setup` passing the network config JSON +3. Netavark creates a bridge (e.g., `podman0`) if it doesn't exist -- default subnet is `10.88.0.0/16` +4. Netavark creates a veth pair -- one end goes into the container's netns, the other attaches to the bridge +5. Netavark assigns an IP from the subnet to the container's veth interface (host-local IPAM) +6. Netavark configures iptables/nftables rules -- masquerade for outbound, DNAT for port mappings +7. Netavark starts aardvark-dns if DNS is enabled, listening on the bridge gateway address + +```text +Host Kernel + | + +-- Bridge interface (e.g., "podman0") <-- created by Netavark + | | + | +-- veth pair endpoint (host side, container 1) + | +-- veth pair endpoint (host side, container 2) + | + +-- Host physical interface (e.g., eth0) + | + +-- NAT (iptables/nftables rules managed by Netavark) +``` + +Netavark also supports macvlan networks (container gets a sub-interface of a physical host NIC with its own MAC, appearing directly on the physical network) and external plugins via a documented JSON API. + +## How Pasta Works (Rootless) + +### The Problem + +Unprivileged users cannot create network interfaces on the host. They cannot create veth pairs, bridges, or configure iptables rules. Netavark's bridge approach cannot work directly for rootless containers. + +### The Solution + +Pasta (part of the `passt` project -- same binary, different command name) operates entirely in userspace, translating between the container's L2 TAP interface and the host's L4 sockets. It requires no capabilities or privileges. + +```text +Container Network Namespace + | + +-- TAP device (e.g., "eth0") + | ^ + | | L2 frames (Ethernet) + | v + +-- pasta process (userspace) + | + | Translation: L2 frames <-> L4 sockets + | + v + Host Network Stack (native TCP/UDP/ICMP sockets) +``` + +### Detailed Data Path + +For an outbound TCP connection from a container: + +1. The application calls `connect()` to an external address +2. The kernel routes the packet through the default gateway to the TAP device +3. Pasta reads the raw Ethernet frame from the TAP file descriptor +4. Pasta parses L2/L3/L4 headers and identifies the TCP SYN +5. Pasta opens a native TCP socket on the host and calls `connect()` to the same destination +6. When the host socket connects, pasta reflects the SYN-ACK back through the TAP as an L2 frame +7. For ongoing data transfer, pasta translates between TAP frames and the host socket, coordinating TCP windows and acknowledgments between the two sides + +Pasta does NOT maintain per-connection packet buffers -- it reflects observed sending windows and ACKs directly between peers. This is a thinner translation layer than a full TCP/IP stack (like slirp4netns used). + +### Built-in Services + +Pasta includes minimalistic network services so the container's stack can auto-configure: + +| Service | Purpose | +|---------|---------| +| ARP proxy | Resolves the gateway address to the host's MAC address | +| DHCP server | Hands out a single IPv4 address (same as host's upstream interface) | +| NDP proxy | Handles IPv6 neighbor discovery, SLAAC prefix advertisement | +| DHCPv6 server | Hands out a single IPv6 address (same as host's upstream interface) | + +By default there is no NAT -- pasta copies the host's IP addresses into the container namespace. + +### Local Connection Bypass (Splice Path) + +For connections between the container and the host, pasta implements a zero-copy bypass: + +- Packets with a local destination skip L2 translation entirely +- `splice(2)` for TCP (zero-copy), `recvmmsg(2)` / `sendmmsg(2)` for UDP (batched) +- Achieves ~38 Gbps TCP throughput for local connections + +### Port Forwarding + +By default, pasta uses auto-detection: it scans `/proc/net/tcp` and `/proc/net/tcp6` periodically and automatically forwards any ports that are bound/listening. Port forwarding is fully configurable via pasta options. + +### Security Properties + +- No dynamic memory allocation (`sbrk`, `brk`, `mmap` blocked via seccomp) +- All capabilities dropped (except `CAP_NET_BIND_SERVICE` if granted) +- Restrictive seccomp profiles (43 syscalls allowed on x86_64) +- Detaches into its own user, mount, IPC, UTS, PID namespaces +- No external dependencies beyond libc +- ~5,000 lines of code target + +### Inter-Container Limitation + +Unlike bridge networking, pasta containers are isolated from each other by default. No virtual bridge connects them. Communication requires port mappings through the host, pods (shared network namespace), or opting into rootless Netavark bridge networking via `podman network create`. + +## Three Nested Namespaces in the Podman Driver + +The Podman compute driver creates three layers of network isolation: + +```text +Namespace 1: Host + | + pasta manages port forwarding (127.0.0.1:) + gateway listens on 0.0.0.0:8080 + | +Namespace 2: Rootless Podman network namespace (managed by pasta) + | + Bridge "openshell" (10.89.x.0/24) + aardvark-dns for container name resolution + | + Container netns (10.89.x.2) + supervisor, proxy, SSH daemon all run here + | +Namespace 3: Inner sandbox netns (created by supervisor) + | + veth pair (10.200.0.1 <-> 10.200.0.2) + iptables forces all traffic through proxy + user workload runs here +``` + +Pasta bridges namespace 1 and 2, the veth pair bridges namespace 2 and 3, and the proxy at the boundary of 2/3 enforces network policy. + +### Layer 1: Pasta (Rootless Podman Bridge) + +At driver startup (`driver.rs:104-114`), the driver ensures a Podman bridge network exists: + +```rust +client.ensure_network(&config.network_name).await?; +``` + +This creates a bridge network named `"openshell"` (default from `DEFAULT_NETWORK_NAME` in `openshell-core/src/config.rs`) with `dns_enabled: true`. In rootless mode, this bridge exists inside a user namespace managed by pasta. The bridge IP range (e.g., `10.89.x.x`) is not routable from the host. + +```text +Host (your machine) + | + 127.0.0.1: <--- pasta binds this on the host + | + [pasta process] <--- translates L4 sockets <-> L2 TAP frames + | + [rootless network namespace] + | + Bridge "openshell" (10.89.1.0/24) + | + +-- 10.89.1.1 (bridge gateway, aardvark-dns listens here) + | + +-- veth --> Container netns + | + 10.89.1.2 (container IP) +``` + +### Layer 2: Container Networking (Pasta Port Forwarding) + +The container spec (`container.rs:447-471`) configures: + +- `nsmode: "bridge"` -- uses the Podman bridge network +- `networks: {"openshell"}` -- attaches to the named bridge +- `portmappings: [{host_port: 0, container_port: 2222, protocol: "tcp"}]` -- publishes SSH on an ephemeral host port +- `hostadd: ["host.containers.internal:host-gateway"]` -- resolves to the host IP (pasta uses `169.254.1.2` in rootless mode) + +Pasta is never explicitly configured. The driver sets `nsmode: "bridge"` and Podman selects pasta automatically as the rootless network backend. The driver logs the detected backend at startup (`driver.rs:86`): + +```rust +network_backend = %info.host.network_backend, +``` + +The `host.containers.internal` hostname (the Podman equivalent of Docker's `host.docker.internal`) is injected into `/etc/hosts` so the supervisor can reach the gateway on the host. The gRPC callback endpoint is auto-detected at `driver.rs:116-130`: + +```rust +if config.grpc_endpoint.is_empty() { + config.grpc_endpoint = + format!("http://host.containers.internal:{}", config.gateway_port); +} +``` + +The bridge gateway IP does NOT work for this purpose in rootless mode because it lives inside the user namespace, not on the host. + +### Layer 3: Inner Sandbox Network Namespace + +Inside the container, the supervisor creates another network namespace (`netns.rs:53-178`, setup at lines 53-63, `ip netns add` at line 77) for the user workload: + +```text +Container (10.89.1.2 on the Podman bridge) + | + [Supervisor process - runs in container's default netns] + | + +-- Proxy listener at 10.200.0.1:3128 + | + +-- veth pair: veth-h-{short_id} <-> veth-s-{short_id} + | + +-- Inner network namespace "sandbox-{short_id}" (short_id = first 8 chars of UUID) + | + 10.200.0.2/24 + | + default route -> 10.200.0.1 (supervisor's proxy) + | + [User's code runs here] + | + iptables rules (IPv4; IPv6 installed best-effort): + ACCEPT -> 10.200.0.1:{proxy_port} TCP (proxy) + ACCEPT -> loopback (-o lo) + ACCEPT -> established/related (conntrack) + LOG -> TCP SYN bypass attempts (rate-limited 5/sec) + REJECT -> TCP (icmp-port-unreachable) + LOG -> UDP bypass attempts (rate-limited 5/sec) + REJECT -> UDP (icmp-port-unreachable) +``` + +The supervisor uses `nsenter --net=` rather than `ip netns exec` to avoid sysfs remount issues that arise under rootless Podman where real `CAP_SYS_ADMIN` is unavailable (`netns.rs:681-716`, function body at 691). + +A tmpfs is mounted at `/run/netns` in the container spec (`container.rs:458-463`) so the supervisor can create named network namespaces. In rootless Podman this directory does not exist on the host, so `mkdir` would fail with `EPERM` without a private tmpfs. + +## Complete Data Paths + +### SSH Session: Client to Sandbox Shell + +```text +Client (CLI on user's machine) + | + 1. gRPC: CreateSshSession -> gateway (returns token, connect_path) + 2. HTTP CONNECT /connect/ssh to gateway + (headers: x-sandbox-id, x-sandbox-token) + | +Gateway (host, port 8080) + | + 3. Looks up SupervisorSession for sandbox_id + 4. Sends RelayOpen{channel_id} over ConnectSupervisor bidi stream + | + [gRPC traverses: host -> pasta L4 translation -> container bridge] + | +Supervisor (inside container at 10.89.x.2) + | + 5. Receives RelayOpen, opens new RelayStream RPC back to gateway + 6. Sends RelayInit{channel_id} on the stream + 7. Connects to Unix socket /run/openshell/ssh.sock + 8. Bidirectional bridge: RelayStream <-> Unix socket (16 KiB chunks) + | +SSH daemon (inside container, Unix socket only, root-only permissions) + | + 9. Authenticates (all auth accepted -- access gated by relay chain) + 10. Spawns shell process + 11. Shell enters inner netns via setns(fd, CLONE_NEWNET) + | +User's shell (in sandbox netns at 10.200.0.2) +``` + +The SSH daemon listens on a Unix socket (not a TCP port) with 0600 permissions. The published port mapping (`host_port: 0 -> container_port: 2222`) exists in the container spec but is currently inert -- nothing listens on TCP 2222 inside the container. All SSH communication uses the gRPC reverse-connect relay pattern exclusively. + +### Outbound HTTP Request from Sandbox Process + +```text +User's code (inner netns, 10.200.0.2) + | + 1. curl https://api.example.com + (HTTP_PROXY=http://10.200.0.1:3128 set via environment) + | + 2. TCP connect to 10.200.0.1:3128 + (allowed by iptables -- only permitted egress destination) + | + 3. HTTP CONNECT api.example.com:443 + | +Supervisor proxy (10.200.0.1:3128 in container netns) + | + 4. OPA policy evaluation (process identity via /proc/net/tcp -> PID) + 5. SSRF check (block internal IPs unless allowed by policy) + 6. Optional L7: TLS intercept, HTTP method/path inspection + | + 7. If allowed: TCP connect to api.example.com:443 + (from container netns, 10.89.x.2) + | + 8. Through Podman bridge -> pasta L2-to-L4 -> host -> internet +``` + +### Supervisor gRPC Callback to Gateway + +The Podman driver auto-detects the callback endpoint scheme based on +whether TLS client certificates are configured. When the RPM's +auto-generated PKI is in place, the endpoint is +`https://host.containers.internal:8080` and the supervisor connects +with mTLS. Without TLS configuration, it falls back to +`http://host.containers.internal:8080`. + +```text +Supervisor (container netns, 10.89.x.2) + | + 1. mTLS connect to https://host.containers.internal:8080 + (resolves to 169.254.1.2:8080 via /etc/hosts) + Client cert bind-mounted from host at /etc/openshell/tls/client/ + | + 2. Routed through container default gateway (bridge) + | + 3. Pasta translates: L2 frame -> host L4 socket + | + 4. Host TCP socket connects to gateway (0.0.0.0:8080) + | +Gateway (host, 0.0.0.0:8080, mTLS enabled) + | + 5. TLS handshake: server presents server cert, client presents client cert + 6. ConnectSupervisor bidirectional stream established + 7. Heartbeats every N seconds (gateway sends interval in SessionAccepted, default 15s) + 8. Reconnects with exponential backoff (1s initial, 30s max) on failure + 9. Same gRPC channel reused for RelayStream calls (no new TLS handshake) +``` + +The gateway binds to `0.0.0.0` by default in the RPM packaging. mTLS +prevents unauthenticated access even though the gateway is reachable +from the network. Client certificates are auto-generated by +`init-pki.sh` on first start and bind-mounted into sandbox containers +by the Podman driver. See `deploy/rpm/CONFIGURATION.md` for the full +configuration reference. + +## Differences from the Kubernetes Driver + +| Aspect | Kubernetes | Podman (rootless pasta) | +|--------|-----------|----------------------| +| Container/Pod IP | Routable cluster-wide | Non-routable (10.89.x.x inside user namespace) | +| Network reachability | Pod IPs reachable from gateway | Bridge not routable from host; requires `host.containers.internal` | +| Sandbox -> Gateway | Direct TCP to K8s service IP | `host.containers.internal` via bridge + pasta | +| SSH transport | Reverse gRPC relay (`ConnectSupervisor` + `RelayStream`) -- same mechanism as Podman | Reverse gRPC relay (`ConnectSupervisor` + `RelayStream`) | +| Port publishing | Not needed (routable IPs) | Ephemeral host port via pasta port forwarding | +| TLS | mTLS via K8s secrets | mTLS via auto-generated PKI (RPM default) or `--disable-tls` | +| DNS | Kubernetes CoreDNS | Podman bridge DNS (aardvark-dns, `dns_enabled: true`) | +| Network policy | K8s NetworkPolicy (ingress restricted to gateway) | iptables inside inner sandbox netns | +| Supervisor delivery | Kubernetes driver managed pod image/template | OCI image volume mount (FROM scratch image) | +| Secrets | K8s Secret volume mount (TLS certs); SSH handshake secret via env var | Podman `secret_env` API (hidden from `podman inspect`) | + +Both drivers use the same reverse gRPC relay (`ConnectSupervisor` + `RelayStream`) for SSH transport. The most significant difference is network reachability: in rootless Podman, the bridge network is not routable from the host, so all communication between host and container goes through either pasta port forwarding (`portmappings`) or the `host.containers.internal` hostname (resolved to `169.254.1.2` by pasta). + +## Port Assignments + +| Port | Component | Purpose | +|------|-----------|---------| +| 8080 | Gateway | gRPC + HTTP multiplexed (default `DEFAULT_SERVER_PORT`) | +| 2222 | Sandbox | Port mapping in container spec (default `DEFAULT_SSH_PORT`); currently inert -- SSH daemon uses Unix socket only | +| 3128 | Sandbox proxy | HTTP CONNECT proxy (inside container, on inner netns host side) | +| 0 (ephemeral) | Host (via pasta) | Published mapping for container SSH port | + +## Key Source Files + +| File | What it controls | +|------|-----------------| +| `crates/openshell-driver-podman/src/driver.rs` | Bridge network creation, gRPC endpoint auto-detection, rootless checks | +| `crates/openshell-driver-podman/src/container.rs` | Container spec: network mode, port mappings, hostadd, tmpfs, capabilities | +| `crates/openshell-driver-podman/src/client.rs` | Podman REST API calls for network ensure/inspect, port discovery | +| `crates/openshell-driver-podman/src/config.rs` | Network name, socket path, SSH port, gateway port defaults | +| `crates/openshell-sandbox/src/sandbox/linux/netns.rs` | Inner network namespace: veth pair, IP addressing, iptables rules | +| `crates/openshell-sandbox/src/proxy.rs` | HTTP CONNECT proxy: OPA policy, SSRF protection, L7 inspection | +| `crates/openshell-sandbox/src/ssh.rs` | SSH daemon on Unix socket, shell process netns entry via `setns()` | +| `crates/openshell-sandbox/src/supervisor_session.rs` | gRPC ConnectSupervisor stream, RelayStream for SSH tunneling | +| `crates/openshell-sandbox/src/grpc_client.rs` | gRPC channel to gateway (mTLS or plaintext, keep-alive, adaptive windowing) | +| `crates/openshell-server/src/ssh_tunnel.rs` | Gateway-side SSH tunnel: HTTP CONNECT endpoint, relay bridging | +| `crates/openshell-server/src/supervisor_session.rs` | SupervisorSessionRegistry, relay claim/open lifecycle | +| `crates/openshell-server/src/compute/mod.rs` | `ComputeRuntime::new_podman()` -- Podman compute driver initialization | +| `crates/openshell-core/src/config.rs` | Default constants: ports, network name | diff --git a/architecture/policy-advisor.md b/architecture/policy-advisor.md index 6d2728332b..c70bfcbd37 100644 --- a/architecture/policy-advisor.md +++ b/architecture/policy-advisor.md @@ -208,6 +208,7 @@ The TUI sandbox screen includes a "Network Rules" panel accessible via `[r]` fro - Expanded detail popup with full binary path, rationale, security notes, and proposed rule Keybindings are state-aware: + - **Pending** → `[a]` approve, `[x]` reject, `[A]` approve all - **Approved** → `[x]` revoke - **Rejected** → `[a]` approve diff --git a/architecture/sandbox-connect.md b/architecture/sandbox-connect.md index 9abb0383a1..cf49ff9169 100644 --- a/architecture/sandbox-connect.md +++ b/architecture/sandbox-connect.md @@ -8,9 +8,27 @@ Sandbox connect provides secure remote access into running sandbox environments. 2. **Command execution** (`sandbox create -- `) -- runs a command over SSH with stdout/stderr piped back 3. **File sync** (`sandbox create --upload`) -- uploads local files into the sandbox before command execution -All three modes tunnel SSH traffic through the gateway's multiplexed port using HTTP CONNECT. The gateway authenticates each connection with a short-lived session token, then performs a custom NSSH1 handshake with the sandbox's embedded SSH daemon before bridging raw bytes between client and sandbox. +Gateway connectivity is **supervisor-initiated**: the gateway never dials the sandbox pod. On startup, each sandbox's supervisor opens a long-lived bidirectional gRPC stream (`ConnectSupervisor`) to the gateway and holds it for the sandbox's lifetime. **`CreateSshSession` → HTTP CONNECT and `ExecSandbox` both depend on that registration**: `open_relay` blocks until a live `ConnectSupervisor` entry exists for the `sandbox_id`; if the supervisor never registers (wrong endpoint, bad env, crash loop), the client hits the supervisor-session wait timeout instead of getting a relay. When a client asks the gateway for SSH, the gateway sends a `RelayOpen` message over that stream; the supervisor responds by initiating a `RelayStream` gRPC call that rides the same TCP+TLS+HTTP/2 connection as a new multiplexed stream. The supervisor bridges the bytes of that stream into a root-owned Unix socket where the embedded SSH daemon listens. **The in-container sshd is reached only on that local Unix socket** — the supervisor `UnixStream::connect`s to it. Do not assume the relay path terminates at a container-exposed TCP listener for sshd; any optional TCP surface is separate from the gateway relay bridge. -There is also a gateway-side `ExecSandbox` gRPC RPC that executes commands inside sandboxes without requiring an external SSH client. This is used for programmatic execution. +There is also a gateway-side `ExecSandbox` gRPC RPC that executes commands inside sandboxes without requiring an external SSH client. It uses the same relay mechanism. + +### Podman and relay environment + +The **Podman** compute driver (`crates/openshell-driver-podman/src/container.rs`, `build_env` / `build_container_spec`) must inject the same **relay-critical** environment variables into the container as the Kubernetes driver: `OPENSHELL_ENDPOINT` (gateway gRPC), `OPENSHELL_SANDBOX_ID`, and `OPENSHELL_SSH_SOCKET_PATH` (Unix path the embedded sshd binds and the supervisor dials). Without `OPENSHELL_SSH_SOCKET_PATH`, the in-container `openshell-sandbox` process does not know where to create the socket; without `OPENSHELL_ENDPOINT` / `OPENSHELL_SANDBOX_ID`, the supervisor cannot complete `ConnectSupervisor`, so the gateway never has a session to target with `RelayOpen`. Driver-owned keys overwrite user spec/template env so these cannot be overridden. **Podman container readiness** (libpod `HealthConfig` in `build_container_spec`) treats the sandbox as ready when a sentinel file exists, **or** `test -S` passes on the configured `sandbox_ssh_socket_path` (**supervisor / Unix-socket path**), **or** a legacy TCP listen check on the published SSH port — so the `Ready` phase used by `CreateSshSession` and the SSH tunnel can reflect Unix-socket–based startup, not only a TCP listener. + +## Two-Plane Architecture + +The supervisor and gateway maintain two logical planes over **one TCP+TLS connection**, multiplexed by HTTP/2 streams: + +- **Control plane** -- the `ConnectSupervisor` bidirectional gRPC stream. Carries `SupervisorHello`, heartbeats, `RelayOpen`/`RelayClose` requests from the gateway, and `RelayOpenResult`/`RelayClose` replies from the supervisor. Lives for the lifetime of the sandbox supervisor process. +- **Data plane** -- one `RelayStream` bidirectional gRPC call per SSH connect or exec invocation. Each call is a new HTTP/2 stream on the same connection. Frames are opaque bytes except for the first frame from the supervisor, which is a typed `RelayInit { channel_id }` used to pair the stream with a pending relay slot on the gateway. + +Running both planes over one HTTP/2 connection means each relay avoids a fresh TLS handshake and benefits from a single authenticated transport boundary. Hyper/h2 `adaptive_window(true)` is enabled on both sides so bulk transfers (large file uploads, long exec stdout) aren't pinned to the default 64 KiB stream window. + +The supervisor-initiated direction gives the model two properties: + +1. The sandbox pod exposes no ingress surface. Network reachability is whatever the supervisor itself can reach outward. +2. Authentication reduces to one place: the existing gateway mTLS channel. There is no second application-layer handshake to design, rotate, or replay-protect. ## Components @@ -18,70 +36,126 @@ There is also a gateway-side `ExecSandbox` gRPC RPC that executes commands insid **File**: `crates/openshell-cli/src/ssh.rs` -Contains the client-side SSH and editor-launch helpers for sandbox connectivity: +Client-side SSH and editor-launch helpers: - `sandbox_connect()` -- interactive SSH shell session - `sandbox_exec()` -- non-interactive command execution via SSH -- `sandbox_rsync()` -- file synchronization via rsync over SSH +- `sandbox_rsync()` -- file synchronization via tar-over-SSH - `sandbox_ssh_proxy()` -- the `ProxyCommand` process that bridges stdin/stdout to the gateway -- OpenShell-managed SSH config helpers -- install a single `Include` entry in - `~/.ssh/config` and maintain generated `Host openshell-` blocks in a - separate OpenShell-owned config file for editor workflows +- OpenShell-managed SSH config helpers -- install a single `Include` entry in `~/.ssh/config` and maintain generated `Host openshell-` blocks in a separate OpenShell-owned config file for editor workflows -These are re-exported from `crates/openshell-cli/src/run.rs` for backward compatibility. +Every generated SSH invocation and every entry in the OpenShell-managed `~/.ssh/config` include `ServerAliveInterval=15` and `ServerAliveCountMax=3`. SSH has no other way to observe that the underlying relay (not the end-to-end TCP socket) has silently dropped, so the client falls back to SSH-level keepalives to surface dead connections within ~45 seconds. + +These helpers are re-exported from `crates/openshell-cli/src/run.rs` for backward compatibility. ### CLI `ssh-proxy` subcommand -**File**: `crates/openshell-cli/src/main.rs` (line ~139, `Commands::SshProxy`) +**File**: `crates/openshell-cli/src/main.rs` (`Commands::SshProxy`) -A top-level CLI subcommand (`ssh-proxy`) that the SSH `ProxyCommand` invokes. It receives `--gateway`, `--sandbox-id`, and `--token` flags, then delegates to `sandbox_ssh_proxy()`. This process has no TTY of its own -- it pipes stdin/stdout directly to the gateway tunnel. +A top-level CLI subcommand (`ssh-proxy`) that the SSH `ProxyCommand` invokes. It receives `--gateway`, `--sandbox-id`, `--token`, and `--gateway-name` flags, then delegates to `sandbox_ssh_proxy()`. This process has no TTY of its own -- it pipes stdin/stdout directly to the gateway tunnel. ### gRPC session bootstrap -**Files**: `proto/openshell.proto`, `crates/openshell-server/src/grpc.rs` +**Files**: `proto/openshell.proto`, `crates/openshell-server/src/grpc/sandbox.rs` Two RPCs manage SSH session tokens: -- `CreateSshSession(sandbox_id)` -- validates the sandbox exists and is `Ready`, generates a UUID token, persists an `SshSession` record, and returns the token plus gateway connection details (host, port, scheme, connect path). +- `CreateSshSession(sandbox_id)` -- validates the sandbox exists and is `Ready`, generates a UUID token, persists an `SshSession` record, and returns the token plus gateway connection details (host, port, scheme, connect path, optional TTL). - `RevokeSshSession(token)` -- marks the session's `revoked` flag to `true` in the persistence layer. +### Supervisor session registry + +**File**: `crates/openshell-server/src/supervisor_session.rs` + +`SupervisorSessionRegistry` holds: + +- `sessions: HashMap` -- the active `ConnectSupervisor` stream sender for each sandbox, plus a `session_id` that uniquely identifies each registration. +- `pending_relays: HashMap` -- one entry per `RelayOpen` waiting for the supervisor's `RelayStream` to arrive. + +Key operations: + +- `register(sandbox_id, session_id, tx)` -- inserts a new session and returns the previous sender if it superseded one. Used by `handle_connect_supervisor` to accept a new stream. +- `remove_if_current(sandbox_id, session_id)` -- removes only if the stored `session_id` matches. Guards against the supersede race where an old session's cleanup runs after a newer session has already registered. +- `open_relay(sandbox_id, timeout)` -- called by the gateway tunnel and exec handlers. Waits up to `timeout` for a supervisor session to appear (with exponential backoff 100 ms → 2 s), registers a pending relay slot keyed by a fresh `channel_id`, sends `RelayOpen` to the supervisor, and returns a `oneshot::Receiver` that resolves when the supervisor claims the slot. +- `claim_relay(channel_id)` -- called by `handle_relay_stream` when the supervisor's first `RelayFrame::Init` arrives. Removes the pending entry, enforces a 10-second staleness bound (`RELAY_PENDING_TIMEOUT`), creates a 64 KiB `tokio::io::duplex` pair, hands the gateway-side half to the waiter, and returns the supervisor-side half to be bridged against the inbound/outbound `RelayFrame` streams. +- `reap_expired_relays()` -- bounds leaks from pending slots the supervisor never claimed (e.g., supervisor crashed between `RelayOpen` and `RelayStream`). Scheduled every 30 s by `spawn_relay_reaper()` during server startup. + +The `ConnectSupervisor` handler (`handle_connect_supervisor`) validates `SupervisorHello`, assigns a fresh `session_id`, sends `SessionAccepted { heartbeat_interval_secs: 15 }`, spawns a loop that processes inbound messages (`Heartbeat`, `RelayOpenResult`, `RelayClose`), and emits a `GatewayHeartbeat` every 15 seconds. + +### RelayStream handler + +**File**: `crates/openshell-server/src/supervisor_session.rs` (`handle_relay_stream`) + +Accepts one inbound `RelayFrame` to extract `channel_id` from `RelayInit`, claims the pending relay, then runs two concurrent forwarding tasks: + +- **Supervisor → gateway**: drains `RelayFrame::Data` frames and writes the bytes to the supervisor-side end of the duplex pair. +- **Gateway → supervisor**: reads the duplex in `RELAY_STREAM_CHUNK_SIZE` (16 KiB) chunks and emits `RelayFrame::Data` messages back. + +The first frame that isn't `RelayInit` is rejected (`invalid_argument`). Any non-data frame after init closes the relay. + ### Gateway tunnel handler **File**: `crates/openshell-server/src/ssh_tunnel.rs` An Axum route at `/connect/ssh` on the shared gateway port. Handles HTTP CONNECT requests by: -1. Validating the session token and sandbox readiness -2. Resolving the sandbox pod's network address -3. Opening a TCP connection to the sandbox SSH port -4. Performing the NSSH1 handshake -5. Bridging bytes bidirectionally between the HTTP-upgraded connection and the sandbox TCP stream + +1. Validating the session token (present, not revoked, bound to the sandbox id in `X-Sandbox-Id`, not expired). +2. Confirming the sandbox is in `Ready` phase. +3. Enforcing per-token (max 3) and per-sandbox (max 20) concurrent connection limits. +4. Calling `supervisor_sessions.open_relay(sandbox_id, 30s)` -- the 30-second wait covers the supervisor's initial mTLS + `ConnectSupervisor` handshake on a freshly-scheduled pod. +5. Waiting up to 10 seconds for the supervisor to open its `RelayStream` and deliver the gateway-side `DuplexStream`. +6. Performing the HTTP CONNECT upgrade on the client connection and calling `copy_bidirectional` between the upgraded client socket and the relay stream. + +There is no gateway-to-sandbox TCP dial, handshake preface, or pod-IP resolution in this path. ### Gateway multiplexing **File**: `crates/openshell-server/src/multiplex.rs` -The gateway runs a single listener that multiplexes gRPC and HTTP on the same port. `MultiplexedService` routes based on the `content-type` header: requests with `application/grpc` go to the gRPC router; all others (including HTTP CONNECT) go to the HTTP router. The HTTP router (`crates/openshell-server/src/http.rs`) merges health endpoints with the SSH tunnel router. +The gateway runs a single listener that multiplexes gRPC and HTTP on the same port. `MultiplexedService` routes based on the `content-type` header: requests with `application/grpc` go to the gRPC router; all others (including HTTP CONNECT) go to the HTTP router. The HTTP router (`crates/openshell-server/src/http.rs`) merges health endpoints with the SSH tunnel router. Hyper is configured with `http2().adaptive_window(true)` so the HTTP/2 stream windows grow under load rather than throttling `RelayStream` to the default 64 KiB window. + +### Sandbox supervisor session + +**File**: `crates/openshell-sandbox/src/supervisor_session.rs` + +`spawn(endpoint, sandbox_id, ssh_socket_path)` starts a background task that: + +1. Opens a gRPC `Channel` to the gateway (`http2_adaptive_window(true)`). The same channel multiplexes the control stream and every relay. +2. Sends `SupervisorHello { sandbox_id, instance_id }` as the first outbound message. +3. Waits for `SessionAccepted` (or fails fast on `SessionRejected`). +4. Runs a loop that reads inbound `GatewayMessage` values and emits `SupervisorHeartbeat` at the accepted interval (min 5 s, usually 15 s). +5. On `RelayOpen`, spawns `handle_relay_open()` which opens a new `RelayStream` RPC on the existing channel, sends `RelayInit { channel_id }` as the first frame, dials the local SSH Unix socket, and bridges bytes in both directions in 16 KiB chunks. + +Reconnect policy: the session loop wraps `run_single_session()` with exponential backoff (1 s → 30 s) on any error. A `session_established` / `session_failed` OCSF event is emitted on each attempt. + +The supervisor is a dumb byte bridge with no awareness of the SSH protocol flowing through it. ### Sandbox SSH daemon **File**: `crates/openshell-sandbox/src/ssh.rs` An embedded SSH server built on `russh` that runs inside each sandbox pod. It: -- Generates an ephemeral Ed25519 host key on startup (no persistent key material) -- Validates the NSSH1 handshake preface before starting the SSH protocol -- Accepts any SSH authentication (none or public key) since authorization is handled by the gateway -- Spawns shell processes on a PTY with full sandbox policy enforcement (Landlock, seccomp, network namespace, privilege dropping) -- Supports interactive shells, exec commands, PTY resize, and window change events + +- Generates an ephemeral Ed25519 host key on startup (no persistent key material). +- Listens on a Unix socket (default `/run/openshell/ssh.sock`, see [Unix socket access control](#unix-socket-access-control)). +- Accepts any SSH authentication (none or public key) because authorization is handled upstream by the gateway session token and by filesystem permissions on the socket. +- Spawns shell processes on a PTY with full sandbox policy enforcement (Landlock, seccomp, network namespace, privilege dropping). +- Supports interactive shells, exec commands, PTY resize, window-change events, and loopback-only `direct-tcpip` channels for port forwarding. ### Gateway-side exec (gRPC) -**File**: `crates/openshell-server/src/grpc.rs` (functions `stream_exec_over_ssh`, `start_single_use_ssh_proxy`, `run_exec_with_russh`) +**File**: `crates/openshell-server/src/grpc/sandbox.rs` (`handle_exec_sandbox`, `stream_exec_over_relay`, `start_single_use_ssh_proxy_over_relay`, `run_exec_with_russh`) The `ExecSandbox` gRPC RPC provides programmatic command execution without requiring an external SSH client. It: -1. Spins up a single-use local TCP proxy that performs the NSSH1 handshake -2. Connects a `russh` client through that proxy -3. Authenticates with `none` auth, opens a channel, sends the command -4. Streams stdout/stderr chunks and exit status back to the gRPC caller + +1. Validates `sandbox_id`, `command`, env keys, and field sizes; confirms the sandbox is `Ready`. +2. Calls `supervisor_sessions.open_relay(sandbox_id, 15s)` -- a shorter wait than connect because exec runs in steady state, not on cold start. +3. Waits up to 10 seconds for the relay `DuplexStream` to arrive. +4. Starts a single-use localhost TCP listener on `127.0.0.1:0` and spawns a task that bridges a single accept to the `DuplexStream` with `copy_bidirectional`. This adapts the `DuplexStream` to something `russh::client::connect_stream` can dial. +5. Connects `russh` to the local proxy, authenticates `none` as user `sandbox`, opens a channel, optionally requests a PTY, and executes the shell-escaped command. +6. Streams `stdout`/`stderr`/`exit` events back to the gRPC caller. + +If `timeout_seconds > 0`, the exec is wrapped in `tokio::time::timeout`. On timeout, exit code 124 is sent (matching the `timeout` command convention). ## Connection Flows @@ -93,104 +167,106 @@ The `sandbox connect` command opens an interactive SSH session. sequenceDiagram participant User as User Terminal participant CLI as CLI (sandbox connect) - participant gRPC as Gateway (gRPC) - participant Proxy as CLI (ssh-proxy) - participant GW as Gateway (/connect/ssh) - participant K8s as Pod Resolver - participant SSHD as Sandbox SSH Daemon - - CLI->>gRPC: GetSandbox(name) -> sandbox.id - CLI->>gRPC: CreateSshSession(sandbox_id) - gRPC-->>CLI: token, gateway_host, gateway_port, scheme, connect_path - - Note over CLI: Builds ProxyCommand string
exec()s into ssh process - - User->>Proxy: ssh spawns ProxyCommand subprocess - Proxy->>GW: CONNECT /connect/ssh HTTP/1.1
X-Sandbox-Id, X-Sandbox-Token - GW->>GW: Validate token + sandbox phase - GW->>K8s: Resolve pod IP (or service DNS) - GW->>SSHD: TCP connect to port 2222 - GW->>SSHD: NSSH1 preface (token, ts, nonce, hmac) - SSHD-->>GW: OK - GW-->>Proxy: 200 OK (upgrade) - - Note over Proxy,SSHD: Bidirectional byte stream (SSH protocol) - - Proxy->>SSHD: SSH handshake + auth_none - SSHD-->>Proxy: Auth accepted - Proxy->>SSHD: channel_open + shell_request + participant GW as Gateway + participant Reg as SessionRegistry + participant Sup as Supervisor (sandbox) + participant Sock as SSH Unix socket + participant SSHD as russh daemon + + Note over Sup,GW: On sandbox startup (persistent): + Sup->>GW: ConnectSupervisor stream + SupervisorHello + GW-->>Sup: SessionAccepted{session_id, heartbeat=15s} + + User->>CLI: openshell sandbox connect foo + CLI->>GW: GetSandbox(name) -> sandbox.id + CLI->>GW: CreateSshSession(sandbox_id) + GW-->>CLI: token, gateway_host, gateway_port, scheme, connect_path + + Note over CLI: Builds ProxyCommand string: exec()s ssh + + User->>CLI: ssh spawns ssh-proxy subprocess + CLI->>GW: CONNECT /connect/ssh
X-Sandbox-Id, X-Sandbox-Token + GW->>GW: Validate token + sandbox Ready + GW->>Reg: open_relay(sandbox_id, 30s) + Reg-->>GW: (channel_id, relay_rx) + GW->>Sup: RelayOpen{channel_id} (over ConnectSupervisor) + + Sup->>GW: RelayStream RPC (new HTTP/2 stream) + Sup->>GW: RelayFrame::Init{channel_id} + GW->>Reg: claim_relay(channel_id) -> DuplexStream pair + Reg-->>GW: gateway-side DuplexStream (via relay_rx) + Sup->>Sock: UnixStream::connect(/run/openshell/ssh.sock) + Sock-->>SSHD: connection accepted + + GW-->>CLI: 200 OK (upgrade) + + Note over CLI,SSHD: SSH protocol over:
CLI↔GW (HTTP CONNECT) ↔ RelayStream frames ↔ Sup ↔ Unix socket ↔ SSHD + + CLI->>SSHD: SSH handshake + auth_none + SSHD-->>CLI: Auth accepted + CLI->>SSHD: channel_open + shell_request SSHD->>SSHD: openpty() + spawn /bin/bash -i
(with sandbox policy applied) User<<->>SSHD: Interactive PTY session ``` **Code trace for `sandbox connect`:** -1. `crates/openshell-cli/src/main.rs` -- `SandboxCommands::Connect { name }` dispatches to `run::sandbox_connect()` +1. `crates/openshell-cli/src/main.rs` -- `SandboxCommands::Connect { name }` dispatches to `run::sandbox_connect()`. 2. `crates/openshell-cli/src/ssh.rs` -- `sandbox_connect()` calls `ssh_session_config()`: - - Resolves sandbox name to ID via `GetSandbox` gRPC - - Creates an SSH session via `CreateSshSession` gRPC - - Builds a `ProxyCommand` string: ` ssh-proxy --gateway --sandbox-id --token ` - - If the gateway host is loopback but the cluster endpoint is not, `resolve_ssh_gateway()` overrides the host with the cluster endpoint's host + - Resolves sandbox name to ID via `GetSandbox` gRPC. + - Creates an SSH session via `CreateSshSession` gRPC. + - Builds a `ProxyCommand` string: ` ssh-proxy --gateway --sandbox-id --token --gateway-name `. + - If the SSH gateway host is loopback but the registered gateway endpoint is not, `resolve_ssh_gateway()` overrides the host with the registered endpoint's host. 3. `sandbox_connect()` builds an `ssh` command with: - - `-o ProxyCommand=...` (the proxy command from step 2) + - `-o ProxyCommand=...` - `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o GlobalKnownHostsFile=/dev/null` (ephemeral host keys) + - `-o ServerAliveInterval=15 -o ServerAliveCountMax=3` (surface silently-dropped relays in ~45 s) - `-tt -o RequestTTY=force` (force PTY allocation) - - `-o SetEnv=TERM=xterm-256color` (terminal type) + - `-o SetEnv=TERM=xterm-256color` - `sandbox` as the SSH user -4. If stdin is a terminal (interactive), the CLI calls `exec()` (Unix) to replace itself with the `ssh` process, giving SSH direct terminal ownership. Otherwise it spawns and waits. -5. When SSH starts, it spawns the `ssh-proxy` subprocess as its `ProxyCommand`. -6. `crates/openshell-cli/src/ssh.rs` -- `sandbox_ssh_proxy()`: - - Parses the gateway URL, connects via TCP (plain) or TLS (mTLS) - - Sends a raw HTTP CONNECT request with `X-Sandbox-Id` and `X-Sandbox-Token` headers - - Reads the response status line; proceeds if 200 - - Spawns two `tokio::spawn` tasks for bidirectional copy between stdin/stdout and the gateway stream - - When the remote-to-stdout direction completes, aborts the stdin-to-remote task (SSH has all the data it needs) +4. If stdin is a terminal (interactive), the CLI calls `exec()` (Unix) to replace itself with the `ssh` process. Otherwise it spawns and waits. +5. `sandbox_ssh_proxy()` connects via TCP (plain) or TLS (mTLS) to the gateway, sends a raw HTTP CONNECT request with `X-Sandbox-Id` and `X-Sandbox-Token` headers, and on a 200 response spawns two tasks to copy bytes between stdin/stdout and the tunnel. +6. Gateway-side: `ssh_connect()` in `ssh_tunnel.rs` authorizes the request, opens a relay, waits for the supervisor's `RelayStream`, and bridges the upgraded HTTP connection to the relay with `tokio::io::copy_bidirectional`. +7. Supervisor-side: on `RelayOpen`, `handle_relay_open()` in `crates/openshell-sandbox/src/supervisor_session.rs` opens a `RelayStream` RPC, sends `RelayInit`, dials `/run/openshell/ssh.sock`, and bridges the frames to the Unix socket. ### Command Execution (CLI) The `sandbox exec` path is identical to interactive connect except: -- The SSH command uses `-T -o RequestTTY=no` (no PTY) when `tty=false` -- The command string is passed as the final SSH argument -- The sandbox daemon routes it through `exec_request()` instead of `shell_request()`, spawning `/bin/bash -lc ` + +- The SSH command uses `-T -o RequestTTY=no` (no PTY) when `tty=false`. +- The command string is passed as the final SSH argument. +- The sandbox daemon routes it through `exec_request()` instead of `shell_request()`, spawning `/bin/bash -lc `. When `openshell sandbox create` launches a `--no-keep` command or shell, it keeps the CLI process alive instead of `exec()`-ing into SSH so it can delete the sandbox after SSH exits. The default create flow, along with `--forward`, keeps the sandbox running. ### Port Forwarding (`forward start`) -`openshell forward start ` opens a local SSH tunnel so connections to `127.0.0.1:` -on the host are forwarded to `127.0.0.1:` inside the sandbox. +`openshell forward start ` opens a local SSH tunnel so connections to `127.0.0.1:` on the host are forwarded to `127.0.0.1:` inside the sandbox. Because SSH runs over the same relay as interactive connect, no additional proxying machinery is needed. #### CLI - Reuses the same `ProxyCommand` path as `sandbox connect`. - Invokes OpenSSH with `-N -o ExitOnForwardFailure=yes -L :127.0.0.1: sandbox`. -- By default stays attached in foreground until interrupted (Ctrl+C), and prints an early startup - confirmation after SSH stays up through its initial forward-setup checks. -- With `-d`/`--background`, SSH forks after auth and the CLI exits. The PID is - tracked in `~/.config/openshell/forwards/-.pid` along with sandbox id metadata. +- By default stays attached in foreground until interrupted (Ctrl+C), and prints an early startup confirmation after SSH stays up through its initial forward-setup checks. +- With `-d`/`--background`, SSH forks after auth and the CLI exits. The PID is tracked in `~/.config/openshell/forwards/-.pid` along with sandbox id metadata. - `openshell forward stop ` validates PID ownership and then kills a background forward. - `openshell forward list` shows all tracked forwards. -- `openshell forward stop` and `openshell forward list` are local operations and do not require - resolving an active cluster. -- `openshell sandbox create --forward ` starts a background forward before connect/exec, including - when no trailing command is provided. +- `openshell forward stop` and `openshell forward list` are local operations and do not require resolving an active gateway. +- `openshell sandbox create --forward ` starts a background forward before connect/exec, including when no trailing command is provided. - `openshell sandbox delete` auto-stops any active forwards for the deleted sandbox. #### TUI -The TUI (`crates/openshell-tui/`) supports port forwarding through the create sandbox modal. Users -specify comma-separated ports in the **Ports** field. After sandbox creation: +The TUI (`crates/openshell-tui/`) supports port forwarding through the create sandbox modal. Users specify comma-separated ports in the **Ports** field. After sandbox creation: 1. The TUI polls for `Ready` state (up to 30 attempts at 2-second intervals). 2. Creates an SSH session via `CreateSshSession` gRPC. 3. Spawns background SSH tunnels (`ssh -N -f -L :127.0.0.1:`) for each port. 4. Sends a `ForwardResult` event back to the main loop with the outcome. -Active forwards are displayed in the sandbox table's NOTES column (e.g., `fwd:8080,3000`) and in -the sandbox detail view's Forwards row. +Active forwards are displayed in the sandbox table's NOTES column (e.g., `fwd:8080,3000`) and in the sandbox detail view's Forwards row. -When deleting a sandbox, the TUI calls `stop_forwards_for_sandbox()` before sending the delete -request. PID tracking uses the same `~/.config/openshell/forwards/` directory as the CLI. +When deleting a sandbox, the TUI calls `stop_forwards_for_sandbox()` before sending the delete request. PID tracking uses the same `~/.config/openshell/forwards/` directory as the CLI. #### Shared forward module @@ -202,81 +278,55 @@ Port forwarding PID management and SSH utility functions are shared between the - `save_forward_pid()` / `read_forward_pid()` / `remove_forward_pid()` -- PID file I/O - `list_forwards()` -- lists all active forwards from PID files - `stop_forward()` / `stop_forwards_for_sandbox()` -- kills forwarding processes by PID -- `resolve_ssh_gateway()` -- loopback gateway resolution (see Gateway Loopback Resolution) +- `resolve_ssh_gateway()` -- loopback gateway resolution (see [Gateway Loopback Resolution](#gateway-loopback-resolution)) - `shell_escape()` -- safe shell argument escaping for SSH commands - `build_sandbox_notes()` -- builds notes strings (e.g., `fwd:8080,3000`) from active forwards #### Supervisor `direct-tcpip` handling -The sandbox SSH server (`crates/openshell-sandbox/src/ssh.rs`) implements -`channel_open_direct_tcpip` from the russh `Handler` trait. - -- **Loopback-only**: only `127.0.0.1`, `localhost`, and `::1` destinations are accepted. - Non-loopback destinations are rejected (`Ok(false)`) to prevent the sandbox from being - used as a generic proxy. -- **Bridge**: accepted channels spawn a tokio task that connects a `TcpStream` to the - target address and uses `copy_bidirectional` between the SSH channel stream and the - TCP stream. -- No additional state is stored on `SshHandler` — the `Channel` object from russh is - self-contained, so forwarding channels are fully independent of session channels. +The sandbox SSH server (`crates/openshell-sandbox/src/ssh.rs`) implements `channel_open_direct_tcpip` from the russh `Handler` trait. -#### Flow - -```mermaid -sequenceDiagram - participant App as Local Application - participant SSH as OpenSSH Client - participant GW as Gateway (CONNECT) - participant SSHD as Sandbox SSH - participant SVC as Service in Sandbox - - SSH->>GW: CONNECT /connect/ssh - GW->>SSHD: TCP + Preface handshake - SSH->>SSHD: direct-tcpip channel (127.0.0.1:port) - SSHD->>SVC: TcpStream::connect(127.0.0.1:port) - App->>SSH: connect to 127.0.0.1:port (local) - SSH->>SSHD: channel data - SSHD->>SVC: TCP data - SVC-->>SSHD: TCP response - SSHD-->>SSH: channel data - SSH-->>App: response -``` +- **Loopback-only**: only `127.0.0.1`, `localhost`, and `::1` destinations are accepted. Non-loopback destinations are rejected (`Ok(false)`) to prevent the sandbox from being used as a generic proxy. +- **Bridge**: accepted channels spawn a tokio task that connects a `TcpStream` to the target address and uses `copy_bidirectional` between the SSH channel stream and the TCP stream. ### Gateway-side Exec (gRPC) -The `ExecSandbox` gRPC RPC bypasses the external SSH client entirely. +The `ExecSandbox` gRPC RPC bypasses the external SSH client entirely while using the same relay plumbing. ```mermaid sequenceDiagram participant Client as gRPC Client - participant Server as Gateway (gRPC) - participant Proxy as Single-Use TCP Proxy - participant SSHD as Sandbox SSH Daemon - - Client->>Server: ExecSandbox(sandbox_id, command, stdin, timeout) - Server->>Server: Validate sandbox exists + Ready - Server->>Server: Resolve target host:port - Server->>Proxy: Bind 127.0.0.1:0 (ephemeral port) - Proxy->>SSHD: TCP connect + NSSH1 handshake - SSHD-->>Proxy: OK - - Server->>Proxy: russh client connects to 127.0.0.1: - Proxy<<->>SSHD: Bridge bytes bidirectionally - Server->>SSHD: SSH auth_none + channel_open + exec(command) - Server->>SSHD: stdin payload + EOF + participant GW as Gateway + participant Reg as SessionRegistry + participant Sup as Supervisor + participant SSHD as SSH daemon (Unix socket) + + Client->>GW: ExecSandbox(sandbox_id, command, stdin, timeout) + GW->>GW: Validate sandbox exists + Ready + GW->>Reg: open_relay(sandbox_id, 15s) + Reg-->>GW: (channel_id, relay_rx) + GW->>Sup: RelayOpen{channel_id} + + Sup->>GW: RelayStream + RelayInit{channel_id} + GW->>Reg: claim_relay -> DuplexStream + Sup->>SSHD: connect /run/openshell/ssh.sock + + Note over GW: start_single_use_ssh_proxy_over_relay
(127.0.0.1:ephemeral -> DuplexStream) + + GW->>GW: russh client dials 127.0.0.1: + GW->>SSHD: SSH auth_none + channel_open + exec(command) + GW->>SSHD: stdin payload + EOF loop Stream output - SSHD-->>Server: stdout/stderr chunks - Server-->>Client: ExecSandboxEvent (Stdout/Stderr) + SSHD-->>GW: stdout/stderr chunks + GW-->>Client: ExecSandboxEvent (Stdout/Stderr) end - SSHD-->>Server: ExitStatus - Server-->>Client: ExecSandboxEvent (Exit) + SSHD-->>GW: ExitStatus + GW-->>Client: ExecSandboxEvent (Exit) ``` -The `start_single_use_ssh_proxy()` function creates a one-shot TCP listener on localhost, accepts a single connection, performs the NSSH1 handshake with the sandbox, then bridges bytes. The `run_exec_with_russh()` function connects through this local proxy, authenticates, executes the command, and streams channel messages to the gRPC response stream. - -If `timeout_seconds > 0`, the exec is wrapped in `tokio::time::timeout`. On timeout, exit code 124 is sent (matching the `timeout` command convention). +`start_single_use_ssh_proxy_over_relay()` exists only as an adapter so `russh::client::connect_stream` can consume the relay `DuplexStream` through an ephemeral TCP listener on `127.0.0.1:0`. It never reaches the network. ### File Sync @@ -288,10 +338,10 @@ File sync uses **tar-over-SSH**: the CLI streams a tar archive through the exist When `--upload` is passed to `sandbox create`, the CLI pushes local files into `/sandbox` (or a specified destination) after the sandbox reaches `Ready` and before any command runs. -1. `git_repo_root()` determines the repository root via `git rev-parse --show-toplevel` -2. `git_sync_files()` lists files with `git ls-files -co --exclude-standard -z` (tracked + untracked, respecting gitignore, null-delimited) -3. `sandbox_sync_up_files()` creates an SSH session config, spawns `ssh sandbox "tar xf - -C /sandbox"`, and streams a tar archive of the file list to the SSH child's stdin using the `tar` crate -4. Files land in `/sandbox` inside the container +1. `git_repo_root()` determines the repository root via `git rev-parse --show-toplevel`. +2. `git_sync_files()` lists files with `git ls-files -co --exclude-standard -z` (tracked + untracked, respecting gitignore, null-delimited). +3. `sandbox_sync_up_files()` creates an SSH session config, spawns `ssh sandbox "tar xf - -C /sandbox"`, and streams a tar archive of the file list to the SSH child's stdin using the `tar` crate. +4. Files land in `/sandbox` inside the container. #### `openshell sandbox upload` / `openshell sandbox download` @@ -306,152 +356,98 @@ openshell sandbox download [] ``` - **Upload**: `sandbox_upload()` streams a tar archive of the local path to `ssh ... tar xf - -C ` on the sandbox side. Default destination: `/sandbox`. + Named directory uploads preserve the source directory basename at the destination, matching `scp -r` and `cp -r`; uploading `.` remains flat. + `.gitignore` filtering only changes which files are included, not the destination layout. - **Download**: `sandbox_download()` runs `ssh ... tar cf - -C ` on the sandbox side and extracts the output locally via `tar::Archive`. Default destination: `.` (current directory). -- No compression for v1 — the SSH tunnel is local-network; compression adds CPU cost with marginal bandwidth savings. +- No compression for v1 -- the SSH tunnel rides the already-TLS-encrypted gateway connection; compression adds CPU cost with marginal bandwidth savings. -#### Why tar-over-SSH instead of rsync - -| | tar-over-SSH | rsync | -|---|---|---| -| **Client dependency** | None — `tar` crate is compiled into the CLI | Requires `rsync` installed on the client machine | -| **Sandbox dependency** | GNU `tar` (present in every base image) | Requires `rsync` installed in the container | -| **Bidirectional** | Same pipe pattern reversed for push/pull | Needs different invocation or rsync daemon for pull | -| **Transport complexity** | Single process (`ssh ... tar xf -`) | Two processes coordinating a delta-transfer protocol through the proxy tunnel | -| **Incremental sync** | Re-sends everything every time | Only transfers changed blocks (faster for repeated syncs of large repos) | -| **Compression** | Uncompressed (can add gzip via `flate2` later) | Built-in `-z` flag | +## Supervisor Session Lifecycle -For OpenShell's use case — one-shot or on-demand pushes of project files over a local network tunnel — the incremental sync advantage of rsync is marginal. Eliminating the external dependency and getting clean bidirectional support outweigh the delta-transfer benefit. If repeated rapid re-syncs of large repos become a need (e.g., a watch mode), revisit by adding content-hash-based skip lists or gzip compression. +Each sandbox has at most one live `ConnectSupervisor` stream at a time. The registry enforces this via `register()`, which overwrites any previous entry. -## NSSH1 Handshake Protocol +### States -The NSSH1 ("OpenShell SSH v1") handshake authenticates the gateway to the sandbox daemon, preventing direct pod access from outside the gateway. - -### Wire Format - -A single newline-terminated text line: - -``` -NSSH1 \n +```mermaid +stateDiagram-v2 + [*] --> Connecting: spawn() + Connecting --> Rejected: SessionRejected + Connecting --> Live: SessionAccepted + Live --> Live: Heartbeats
RelayOpen/Result
RelayClose + Live --> Disconnected: stream closed / error + Disconnected --> Connecting: backoff (1s..30s) + Rejected --> Connecting: backoff (1s..30s) + Live --> [*]: sandbox exits ``` -| Field | Type | Description | -|-------------|--------|-------------| -| `NSSH1` | string | Magic prefix (protocol version identifier) | -| `token` | string | UUID session token (from `CreateSshSession` for interactive; freshly generated for gateway-side exec) | -| `timestamp` | i64 | Unix epoch seconds at time of generation | -| `nonce` | string | UUID v4, unique per handshake attempt | -| `hmac` | string | Hex-encoded HMAC-SHA256 of `token\|timestamp\|nonce` keyed on the shared secret | - -### Validation (sandbox side) +### Hello and accept -**File**: `crates/openshell-sandbox/src/ssh.rs` -- `verify_preface()` +The supervisor sends `SupervisorHello { sandbox_id, instance_id }` (where `instance_id` is a fresh UUID per process start) as the first message. The gateway: -1. Split line on whitespace; reject if not exactly 5 fields or magic is not `NSSH1` -2. Parse timestamp; compute absolute clock skew `|now - timestamp|` -3. Reject if skew exceeds `ssh_handshake_skew_secs` (default: 300 seconds) -4. Recompute HMAC-SHA256 over `token|timestamp|nonce` with the shared secret -5. Compare computed signature against the received signature (constant-time via `hmac` crate) -6. Check nonce against the replay cache; reject if the nonce has been seen before within the skew window -7. Insert the nonce into the replay cache on success -8. Respond with `OK\n` on success or `ERR\n` on failure +1. Assigns `session_id = Uuid::new_v4()`. +2. Registers the session; any existing entry is evicted and its sender is dropped. +3. Replies with `SessionAccepted { session_id, heartbeat_interval_secs: 15 }`. +4. Spawns `run_session_loop` to process inbound messages and emit gateway heartbeats. -### Nonce replay detection +On any registration failure (e.g., the supervisor's mpsc receiver was already dropped), `remove_if_current` is called with the assigned `session_id` so the cleanup does not evict a newer successful registration. -The SSH server maintains a per-process `NonceCache` (`HashMap` behind `Arc>`) that tracks nonces seen within the handshake skew window. A background tokio task reaps expired entries every 60 seconds. If a valid preface is presented with a previously-seen nonce, the handshake is rejected. This prevents replay attacks within the timestamp validity window. +### Heartbeats -### HMAC computation +Both directions emit heartbeats at the negotiated interval (15 s). Heartbeats are strictly informational -- their purpose is to keep the HTTP/2 connection warm and let each side detect a half-open transport quickly. There is no explicit application-level timeout that kills the session if heartbeats stop; failures are detected when a send fails or when the stream reports EOF / error. -Both the gateway (`crates/openshell-server/src/ssh_tunnel.rs` -- `build_preface()`) and the gRPC exec path (`crates/openshell-server/src/grpc.rs` -- `build_preface()`) use identical logic: - -```rust -let payload = format!("{token}|{timestamp}|{nonce}"); -let signature = hmac_sha256(secret.as_bytes(), payload.as_bytes()); -// hmac_sha256 returns hex::encode(Hmac::::finalize()) -``` +### Supersede semantics -### Read-line safety +If a supervisor restarts (or a network blip forces a new `ConnectSupervisor` call), the gateway sees a second `SupervisorHello` for the same `sandbox_id`. `register()` inserts the new session and returns the old `tx`. The old session's `run_session_loop` continues to poll its inbound stream until it errors out, at which point its cleanup calls `remove_if_current(sandbox_id, old_session_id)` -- which does nothing because the stored entry now has the new `session_id`. The newer session stays live. -Both sides cap the preface line at 1024 bytes and stop reading at `\n` or EOF. This prevents a misbehaving peer from consuming unbounded memory. +Tests in `supervisor_session.rs` pin this behavior: -## Sandbox SSH Daemon Internals +- `registry_supersedes_previous_session` -- confirms that `register()` returns the prior sender. +- `remove_if_current_ignores_stale_session_id` -- confirms a late cleanup does not evict a newer registration. +- `open_relay_uses_newest_session_after_supersede` -- confirms `RelayOpen` is delivered to the newest session only. -### Startup +### Pending-relay reaper -`run_ssh_server()` in `crates/openshell-sandbox/src/ssh.rs`: +`spawn_relay_reaper(state, 30s)` sweeps `pending_relays` every 30 seconds and removes entries older than `RELAY_PENDING_TIMEOUT` (10 s). This bounds the leak if a supervisor acknowledges `RelayOpen` but crashes before initiating `RelayStream`. -1. Generates an ephemeral Ed25519 host key using `OsRng` -2. Configures `russh::server::Config` with 1-second auth rejection delay -3. Binds a `TcpListener` on the configured address (default: `0.0.0.0:2222`) -4. Enters an accept loop; each connection is handled in a `tokio::spawn` task - -### Connection handling +## Authentication and Security Model -`handle_connection()`: +### Transport authentication -1. Reads and validates the NSSH1 preface (rejects with `ERR\n` on failure) -2. Responds `OK\n` on success -3. Hands the TCP stream to `russh::server::run_stream()` with an `SshHandler` +All gRPC traffic (control plane + data plane + other RPCs) rides one mTLS-authenticated TCP+TLS+HTTP/2 connection from the supervisor to the gateway. Client certificates prove the supervisor's identity; the server certificate proves the gateway's. Nothing sits between the supervisor and the SSH daemon except the Unix socket's filesystem permissions. -### Authentication +The CLI continues to authenticate to the gateway with its own mTLS credentials (or Cloudflare bearer token in reverse-proxy deployments) and a per-session token returned by `CreateSshSession`. The session token is enforced at the gateway: token scope (sandbox id), revocation state, and optional expiry are all checked in `ssh_connect()` before `open_relay()` is called. -The `SshHandler` implements `russh::server::Handler`: +### Unix socket access control -- `auth_none()` returns `Auth::Accept` -- any user is accepted -- `auth_publickey()` returns `Auth::Accept` -- any key is accepted +The supervisor creates `/run/openshell/ssh.sock` (path is configurable via the gateway's `sandbox_ssh_socket_path` / supervisor's `--ssh-socket-path` / `OPENSHELL_SSH_SOCKET_PATH`) and: -Authorization is performed by the gateway (token validation + sandbox readiness check) before the SSH protocol starts. The NSSH1 handshake proves the connection came through an authorized gateway. +1. Creates the parent directory if missing and sets it to mode `0700` (root-owned). +2. Removes any stale socket from a previous run. +3. Binds a `UnixListener` on the path. +4. Sets the socket to mode `0600`. -### Shell and exec +The supervisor runs as root; the sandbox workload runs as an unprivileged user. Only the supervisor can connect to the socket. The workload inside the sandbox has no filesystem path by which it can reach the SSH daemon directly. All ingress goes through the relay bridge, which only the supervisor can open (because only the supervisor holds the gateway session). -- `shell_request()` calls `start_shell(channel, handle, None)` -- spawns `/bin/bash -i` -- `exec_request()` calls `start_shell(channel, handle, Some(command))` -- spawns `/bin/bash -lc ` -- `pty_request()` stores the PTY dimensions for use when spawning the shell -- `window_change_request()` calls `TIOCSWINSZ` ioctl on the PTY master fd +`handle_connection()` in `crates/openshell-sandbox/src/ssh.rs` hands the Unix stream directly to `russh::server::run_stream` with no preface or handshake layer in between. -### PTY and process management +### Kubernetes NetworkPolicy -`spawn_pty_shell()`: +The sandbox pod needs no gateway-to-sandbox ingress rule; the SSH daemon has no TCP listener. Helm ships an egress policy that constrains what the pod can reach outward -- see [Gateway Security](gateway-security.md). -1. Calls `nix::pty::openpty()` with the requested window size -2. Clones the master fd for reading and writing -3. Configures the shell command with environment variables: - - `OPENSHELL_SANDBOX=1`, `HOME=/sandbox`, `USER=sandbox`, `TERM=` - - Proxy vars: `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `NO_PROXY=127.0.0.1,localhost,::1`, `http_proxy`, `https_proxy`, `grpc_proxy`, `no_proxy=127.0.0.1,localhost,::1`, `NODE_USE_ENV_PROXY=1` so Node.js `fetch` honors the proxy env while localhost stays direct - - TLS trust vars: `NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE` - - Provider credential env vars (from the provider registry) -4. Installs a `pre_exec` hook that: - - Calls `setsid()` to create a new session - - Calls `TIOCSCTTY` to set the slave PTY as the controlling terminal - - Enters the network namespace (`setns(fd, CLONE_NEWNET)`) if configured (Linux only) - - Drops privileges (`initgroups` + `setgid` + `setuid`) per the sandbox policy - - Applies sandbox restrictions (Landlock, seccomp) via `sandbox::apply()` -5. Spawns the child process +### What SSH auth does NOT enforce -### I/O threading +The embedded SSH daemon accepts all authentication attempts. This is intentional: -Three threads handle the PTY I/O: +- The gateway already validated the session token and sandbox readiness. +- Unix socket permissions already restrict who can connect to the daemon to the supervisor, and the supervisor only opens the socket in response to a gateway `RelayOpen`. +- SSH key management would add complexity without additional security value in this architecture. -1. **Writer thread** (std::thread) -- receives bytes from `SshHandler::data()` via an `mpsc::channel` and writes them to the PTY master -2. **Reader thread** (std::thread) -- reads from PTY master in 4096-byte chunks, dispatches each chunk to the SSH channel via `handle.data()` on the tokio runtime. Sends EOF when the master returns 0 or errors. Signals completion via a `reader_done_tx` channel. -3. **Exit thread** (std::thread) -- waits for `child.wait()`, then waits for the reader thread to finish (via `reader_done_rx`), then sends `exit_status_request` and `close` on the SSH channel +### Ephemeral host keys -The reader-done synchronization ensures correct SSH protocol ordering: data -> EOF -> exit-status -> close. +The sandbox generates a fresh Ed25519 host key on every startup. The CLI disables `StrictHostKeyChecking` and sets `UserKnownHostsFile=/dev/null` and `GlobalKnownHostsFile=/dev/null` to avoid known-hosts conflicts. ## Sandbox Target Resolution -The gateway and the gRPC exec path both resolve the sandbox's network address using the same logic. - -**File**: `crates/openshell-server/src/ssh_tunnel.rs` (gateway), `crates/openshell-server/src/grpc.rs` (exec) - -Resolution order: -1. If the sandbox has a `status.agent_pod` field, resolve the pod IP via the Kubernetes API (`agent_pod_ip()`) -2. Otherwise, construct a cluster-internal DNS name: `..svc.cluster.local` - -The target port is always `config.sandbox_ssh_port` (default: 2222). - -The `ConnectTarget` enum in `ssh_tunnel.rs` encodes both cases: -- `ConnectTarget::Ip(SocketAddr)` -- direct IP from pod resolution -- `ConnectTarget::Host(String, u16)` -- DNS hostname fallback +The gateway does not resolve a sandbox's network address or port. The only identifier that matters is `sandbox_id`, which keys into the supervisor session registry. ## API and Persistence @@ -460,9 +456,11 @@ The `ConnectTarget` enum in `ssh_tunnel.rs` encodes both cases: **Proto**: `proto/openshell.proto` -- `CreateSshSessionRequest` / `CreateSshSessionResponse` Request: + - `sandbox_id` (string) -- the sandbox to connect to Response: + - `sandbox_id` (string) - `token` (string) -- UUID session token - `gateway_host` (string) -- resolved from `Config::ssh_gateway_host` (defaults to bind address if empty) @@ -470,13 +468,16 @@ Response: - `gateway_scheme` (string) -- `"https"` if TLS is configured, otherwise `"http"` - `connect_path` (string) -- from `Config::ssh_connect_path` (default: `/connect/ssh`) - `host_key_fingerprint` (string) -- currently unused (empty) +- `expires_at_ms` (int64) -- session expiry; 0 disables expiry ### RevokeSshSession Request: + - `token` (string) -- session token to revoke Response: + - `revoked` (bool) -- true if a session was found and revoked ### SshSession persistence @@ -493,128 +494,156 @@ Stored in the gateway's persistence layer (SQLite or Postgres) as object type `" | `created_at_ms` | int64 | Creation time (ms since epoch) | | `revoked` | bool | Whether the session has been revoked | | `name` | string | Auto-generated human-friendly name | +| `expires_at_ms` | int64 | Expiry timestamp; 0 means no expiry | + +A background reaper (`spawn_session_reaper`) deletes revoked and expired rows every hour. + +### ConnectSupervisor / RelayStream + +**Proto**: `proto/openshell.proto` + +- `ConnectSupervisor(stream SupervisorMessage) returns (stream GatewayMessage)` +- `RelayStream(stream RelayFrame) returns (stream RelayFrame)` + +Key messages: + +| Message | Direction | Fields | +|---|---|---| +| `SupervisorHello` | sup → gw | `sandbox_id`, `instance_id` | +| `SessionAccepted` | gw → sup | `session_id`, `heartbeat_interval_secs` | +| `SessionRejected` | gw → sup | `reason` | +| `SupervisorHeartbeat` | sup → gw | (empty) | +| `GatewayHeartbeat` | gw → sup | (empty) | +| `RelayOpen` | gw → sup | `channel_id` (UUID) | +| `RelayOpenResult` | sup → gw | `channel_id`, `success`, `error` | +| `RelayClose` | either | `channel_id`, `reason` | +| `RelayInit` | sup → gw (first `RelayFrame`) | `channel_id` | +| `RelayFrame` | either | `oneof { RelayInit init, bytes data }` | ### ExecSandbox **Proto**: `proto/openshell.proto` -- `ExecSandboxRequest` / `ExecSandboxEvent` Request: + - `sandbox_id` (string) - `command` (repeated string) -- command and arguments - `workdir` (string) -- optional working directory - `environment` (map) -- optional env var overrides (keys validated against `^[A-Za-z_][A-Za-z0-9_]*$`) - `timeout_seconds` (uint32) -- 0 means no timeout - `stdin` (bytes) -- optional stdin payload +- `tty` (bool) -- request a PTY Response stream (`ExecSandboxEvent`): + - `Stdout(data)` -- stdout chunk - `Stderr(data)` -- stderr chunk - `Exit(exit_code)` -- final exit status (124 on timeout) -The gateway builds the remote command by shell-escaping arguments, prepending sorted env var assignments, and optionally wrapping in `cd && ...`. +The gateway builds the remote command by shell-escaping arguments, prepending sorted env var assignments, and optionally wrapping in `cd && ...`. The assembled command is capped at 256 KiB. ## Gateway Loopback Resolution **File**: `crates/openshell-core/src/forward.rs` -- `resolve_ssh_gateway()` -When the gateway returns a loopback address (`127.0.0.1`, `0.0.0.0`, `localhost`, or `::1`), the client overrides it with the host from the cluster endpoint URL. This handles the common case where the gateway defaults to `127.0.0.1` but the cluster is running on a remote machine. +When the gateway returns a loopback address (`127.0.0.1`, `0.0.0.0`, `localhost`, or `::1`), the client overrides it with the host from the registered gateway endpoint URL. This handles the common case where the gateway defaults to `127.0.0.1` but the gateway is running on a remote machine. -The override only applies if the cluster endpoint itself is not also a loopback address. If both are loopback, the original address is kept. +The override only applies if the registered gateway endpoint itself is not also a loopback address. If both are loopback, the original address is kept. This function is shared between the CLI and TUI via the `openshell-core::forward` module. -## Authentication and Security Model - -### Layered authentication - -1. **mTLS (transport layer)** -- when TLS is configured, the CLI authenticates to the gateway using client certificates. The `ssh-proxy` subprocess inherits TLS options from the parent CLI process. -2. **Session token (application layer)** -- the gateway validates the session token against the persistence layer. Tokens are scoped to a specific sandbox and can be revoked. -3. **NSSH1 handshake (gateway-to-sandbox)** -- the shared handshake secret proves the connection originated from an authorized gateway. The timestamp + nonce prevent replay attacks within the skew window. The nonce replay cache rejects duplicates. -4. **Kubernetes NetworkPolicy** -- a Helm-managed `NetworkPolicy` restricts ingress to sandbox pods on port 2222 to only the gateway pod, preventing lateral movement from other in-cluster workloads. Controlled by `networkPolicy.enabled` in the Helm values (default: `true`). - -### Mandatory handshake secret - -The NSSH1 handshake secret (`OPENSHELL_SSH_HANDSHAKE_SECRET`) is required. Both the server and sandbox will refuse to start if the secret is empty or unset. For cluster deployments the secret is auto-generated by the entrypoint script (`deploy/docker/cluster-entrypoint.sh`) via `openssl rand -hex 32` and injected into the Helm values. - -### What SSH auth does NOT enforce - -The embedded SSH daemon accepts all authentication attempts. This is intentional: -- The NSSH1 handshake already proved the connection came through the gateway -- The gateway already validated the session token and sandbox readiness -- SSH key management would add complexity without additional security value in this architecture - -### Ephemeral host keys - -The sandbox generates a fresh Ed25519 host key on every startup. The CLI disables `StrictHostKeyChecking` and sets `UserKnownHostsFile=/dev/null` and `GlobalKnownHostsFile=/dev/null` to avoid known-hosts conflicts. - -## Configuration Reference +## Timeouts -### Gateway configuration - -**File**: `crates/openshell-core/src/config.rs` -- `Config` struct - -| Field | Default | Description | -|----------------------------|------------------|-------------| -| `ssh_gateway_host` | `127.0.0.1` | Public hostname/IP for gateway connections | -| `ssh_gateway_port` | `8080` | Public port for gateway connections (0 = use bind port) | -| `ssh_connect_path` | `/connect/ssh` | HTTP path for CONNECT requests | -| `sandbox_ssh_port` | `2222` | SSH listen port inside sandbox pods | -| `ssh_handshake_secret` | (required) | Shared HMAC key for NSSH1 handshake (server fails to start if empty) | -| `ssh_handshake_skew_secs` | `300` | Maximum allowed clock skew (seconds) | - -### Sandbox environment variables - -These are injected into sandbox pods by the gateway: - -| Variable | Description | -|--------------------------------------|-------------| -| `OPENSHELL_SSH_LISTEN_ADDR` | Address for the embedded SSH server to bind | -| `OPENSHELL_SSH_HANDSHAKE_SECRET` | Shared secret for NSSH1 handshake validation | -| `OPENSHELL_SSH_HANDSHAKE_SKEW_SECS` | Allowed clock skew for handshake timestamp | - -### CLI TLS options - -| Flag / Env Var | Description | -|-----------------------------|-------------| -| `--tls-ca` / `OPENSHELL_TLS_CA` | CA certificate for gateway verification | -| `--tls-cert` / `OPENSHELL_TLS_CERT` | Client certificate for mTLS | -| `--tls-key` / `OPENSHELL_TLS_KEY` | Client private key for mTLS | +| Stage | Duration | Where | +|---|---|---| +| Supervisor session wait (SSH connect) | 30 s | `ssh_tunnel::ssh_connect` -> `open_relay` | +| Supervisor session wait (ExecSandbox) | 15 s | `handle_exec_sandbox` -> `open_relay` | +| Wait for supervisor to claim relay | 10 s | `relay_rx` wrapped in `tokio::time::timeout` | +| Pending-relay TTL (reaper) | 10 s | `RELAY_PENDING_TIMEOUT` in registry | +| Session-wait backoff | 100 ms → 2 s | `wait_for_session` | +| Supervisor reconnect backoff | 1 s → 30 s | `run_session_loop` in sandbox supervisor | +| SSH-level keepalive | 15 s × 3 | CLI / managed ssh-config | +| Supervisor heartbeat | 15 s | `HEARTBEAT_INTERVAL_SECS` | +| SSH session reaper sweep | 1 h | `spawn_session_reaper` | +| Pending-relay reaper sweep | 30 s | `spawn_relay_reaper` | ## Failure Modes | Scenario | Status / Behavior | Source | -|----------|-------------------|--------| +|---|---|---| | Missing `X-Sandbox-Id` or `X-Sandbox-Token` header | `401 Unauthorized` | `ssh_tunnel.rs` -- `header_value()` | | Empty header value | `400 Bad Request` | `ssh_tunnel.rs` -- `header_value()` | | Non-CONNECT method on `/connect/ssh` | `405 Method Not Allowed` | `ssh_tunnel.rs` -- `ssh_connect()` | | Token not found in persistence | `401 Unauthorized` | `ssh_tunnel.rs` -- `ssh_connect()` | | Token revoked or sandbox ID mismatch | `401 Unauthorized` | `ssh_tunnel.rs` -- `ssh_connect()` | +| Token expired | `401 Unauthorized` | `ssh_tunnel.rs` -- `ssh_connect()` | | Sandbox not found | `404 Not Found` | `ssh_tunnel.rs` -- `ssh_connect()` | | Sandbox not in `Ready` phase | `412 Precondition Failed` | `ssh_tunnel.rs` -- `ssh_connect()` | -| Pod IP resolution fails | `502 Bad Gateway` | `ssh_tunnel.rs` -- `ssh_connect()` | -| No pod IP and no sandbox name | `412 Precondition Failed` | `ssh_tunnel.rs` -- `ssh_connect()` | -| Persistence read error | `500 Internal Server Error` | `ssh_tunnel.rs` -- `ssh_connect()` | -| NSSH1 handshake rejected by sandbox | Tunnel closed; `"sandbox handshake rejected"` logged | `ssh_tunnel.rs` -- `handle_tunnel()` | -| HTTP upgrade failure | `"SSH upgrade failed"` logged; tunnel not established | `ssh_tunnel.rs` -- `ssh_connect()` | -| TCP connection to sandbox fails | Tunnel error logged and closed | `ssh_tunnel.rs` -- `handle_tunnel()` | -| SSH exec timeout | Exit code 124 returned | `grpc.rs` -- `stream_exec_over_ssh()` | +| Per-token or per-sandbox concurrency limit hit | `429 Too Many Requests` | `ssh_tunnel.rs` -- `ssh_connect()` | +| Supervisor session not connected after 30 s | `502 Bad Gateway` | `ssh_tunnel.rs` -- `ssh_connect()` | +| Supervisor failed to claim relay within 10 s | Tunnel closed; `"relay open timed out"` logged | `ssh_tunnel.rs` -- spawned tunnel task | +| Relay channel oneshot dropped | Tunnel closed; `"relay channel dropped"` logged | `ssh_tunnel.rs` -- spawned tunnel task | +| First `RelayFrame` not `RelayInit` or empty `channel_id` | `invalid_argument` on `RelayStream` | `supervisor_session.rs` -- `handle_relay_stream` | +| `RelayStream` arrives after pending entry expired (>10 s) | `deadline_exceeded` | `supervisor_session.rs` -- `claim_relay` | +| Gateway restart during live relay | CLI SSH detects via keepalive within ~45 s; relays are torn down with the TCP connection | CLI `ServerAliveInterval=15`, `ServerAliveCountMax=3` | +| Supervisor restart | Gateway sends on stale mpsc fails; client sees same behavior as gateway restart; supervisor's reconnect loop re-registers | `run_session_loop`, `open_relay` | +| Silently-dropped relay (half-open TCP) | CLI-side SSH keepalives probe every 15 s; session exits with `Broken pipe` after 3 missed probes | SSH client keepalives | +| ExecSandbox timeout | Exit code 124 returned to caller | `stream_exec_over_relay` | +| Command exceeds 256 KiB assembled length | `invalid_argument` | `build_remote_exec_command` | ## Graceful Shutdown ### Gateway tunnel teardown -After `copy_bidirectional` completes (either side closes), `handle_tunnel()` calls `AsyncWriteExt::shutdown()` on the upgraded connection to send a clean EOF to the client. This avoids TCP RST and gives SSH time to read remaining protocol data (e.g., exit-status) from its buffer. +After `copy_bidirectional` completes on either side, `ssh_connect()` calls `AsyncWriteExt::shutdown()` on the upgraded client connection so SSH sees a clean EOF and can read any remaining protocol data (e.g., exit-status) before exiting. + +### RelayStream teardown + +The `handle_relay_stream` task half-closes the supervisor-side duplex on inbound EOF so the gateway-side reader sees EOF and terminates its own forwarding task. On the supervisor side, `handle_relay_open` does the symmetric shutdown on the Unix socket after inbound EOF, then drops the outbound mpsc so the gateway observes EOF on the response stream too. -### SSH proxy teardown +### Supervisor session teardown -The `sandbox_ssh_proxy()` function spawns two copy tasks. When the remote-to-stdout task completes, the stdin-to-remote task is aborted. This ensures the proxy exits promptly when the SSH session ends without waiting for the user to type something. +When the sandbox exits, the supervisor process ends, the HTTP/2 connection closes, and all multiplexed streams fail with `stream error`. The gateway's `run_session_loop` observes the error, logs `supervisor session: ended`, and calls `remove_if_current` to deregister. Pending relay slots that never got claimed are swept by `reap_expired_relays` within 30 s. ### PTY reader-exit ordering The sandbox SSH daemon's exit thread waits for the reader thread to finish forwarding all PTY output before sending `exit_status_request` and `close`. This prevents a race where the channel closes before all output has been delivered. +## Configuration Reference + +### Gateway configuration + +**File**: `crates/openshell-core/src/config.rs` -- `Config` struct + +| Field | Default | Description | +|---|---|---| +| `ssh_gateway_host` | `127.0.0.1` | Public hostname/IP advertised in `CreateSshSessionResponse` | +| `ssh_gateway_port` | `8080` | Public port for gateway connections (0 = use bind port) | +| `ssh_connect_path` | `/connect/ssh` | HTTP path for CONNECT requests | +| `sandbox_ssh_socket_path` | `/run/openshell/ssh.sock` | Path the supervisor binds its Unix socket on; passed to the sandbox as `OPENSHELL_SSH_SOCKET_PATH` | +| `ssh_session_ttl_secs` | (default in code) | Default TTL applied to new `SshSession` rows; 0 disables expiry | + +### Sandbox environment variables + +These are injected into compute-backed sandboxes by the **Kubernetes** driver (`crates/openshell-driver-kubernetes/src/driver.rs`), the **Podman** driver (`crates/openshell-driver-podman/src/container.rs`), and the **Docker** driver (`crates/openshell-driver-docker/src/lib.rs`). Together they are required for **persistent `ConnectSupervisor` registration and relay** (see [Podman and relay environment](#podman-and-relay-environment) for the Podman-specific fix): + +| Variable | Description | +|---|---| +| `OPENSHELL_SSH_SOCKET_PATH` | Filesystem path for the embedded SSH server's Unix socket (default `/run/openshell/ssh.sock`); must align with gateway `sandbox_ssh_socket_path` | +| `OPENSHELL_ENDPOINT` | Gateway gRPC endpoint; the supervisor uses this to open `ConnectSupervisor` | +| `OPENSHELL_SANDBOX_ID` | Identifier reported in `SupervisorHello` | + +### CLI TLS options + +| Flag / Env Var | Description | +|---|---| +| `--tls-ca` / `OPENSHELL_TLS_CA` | CA certificate for gateway verification | +| `--tls-cert` / `OPENSHELL_TLS_CERT` | Client certificate for mTLS | +| `--tls-key` / `OPENSHELL_TLS_KEY` | Client private key for mTLS | + ## Cross-References - [Gateway Architecture](gateway.md) -- gateway multiplexing, persistence layer, gRPC service details +- [Gateway Security](gateway-security.md) -- mTLS, session tokens, network policy - [Sandbox Architecture](sandbox.md) -- sandbox lifecycle, policy enforcement, network isolation, proxy - [Providers](sandbox-providers.md) -- provider credential injection into SSH shell processes diff --git a/architecture/sandbox-custom-containers.md b/architecture/sandbox-custom-containers.md index e44de29dd2..b8303c5de0 100644 --- a/architecture/sandbox-custom-containers.md +++ b/architecture/sandbox-custom-containers.md @@ -9,7 +9,7 @@ The `--from` flag accepts four kinds of input: | Input | Example | Behavior | |-------|---------|----------| | **Community sandbox name** | `--from openclaw` | Resolves to `ghcr.io/nvidia/openshell-community/sandboxes/openclaw:latest` | -| **Dockerfile path** | `--from ./Dockerfile` | Builds the image, pushes it into the cluster, then creates the sandbox | +| **Dockerfile path** | `--from ./Dockerfile` | Builds the image locally, makes it available to the local gateway when needed, then creates the sandbox | | **Directory with Dockerfile** | `--from ./my-sandbox/` | Uses the directory as the build context | | **Full image reference** | `--from myregistry.com/img:tag` | Uses the image directly | @@ -19,8 +19,9 @@ The CLI classifies the value in this order: 1. **Existing file** whose name contains "Dockerfile" (case-insensitive) — treated as a Dockerfile to build. 2. **Existing directory** containing a `Dockerfile` — treated as a build context directory. -3. **Contains `/`, `:`, or `.`** — treated as a full container image reference. -4. **Otherwise** — treated as a community sandbox name, expanded to `{OPENSHELL_COMMUNITY_REGISTRY}/{name}:latest`. +3. **Missing explicit local path** (for example `./Dockerfile`, `../ctx`, or an absolute path) — rejected locally instead of sent to the gateway as an image pull. +4. **Contains `/`, `:`, or `.`** — treated as a full container image reference. +5. **Otherwise** — treated as a community sandbox name, expanded to `{OPENSHELL_COMMUNITY_REGISTRY}/{name}:latest`. The community registry prefix defaults to `ghcr.io/nvidia/openshell-community/sandboxes` and can be overridden with the `OPENSHELL_COMMUNITY_REGISTRY` environment variable. @@ -33,39 +34,41 @@ The community registry prefix defaults to `ghcr.io/nvidia/openshell-community/sa When `--from` points to a Dockerfile or directory, the CLI: 1. Builds the image locally via the Docker daemon (respecting `.dockerignore`). -2. Pushes it into the cluster's containerd runtime using `docker save` / `ctr import`. +2. Makes it available to the local gateway runtime when a managed local gateway is running; otherwise keeps the tag in the host Docker daemon for standalone local drivers. 3. Creates the sandbox with the resulting image tag. +The build step aborts with a clear error if the Docker build stream stays silent for longer than `OPENSHELL_BUILD_NO_PROGRESS_TIMEOUT_SECS` seconds (default 1800). This is a guard against deadlocked container runtimes — most commonly an under-provisioned VM (e.g. macOS Colima with the default 2 vCPU / 2 GiB) where BuildKit can stop emitting events partway through a multi-step build and never recover. Raise the value if a legitimate build step is just quiet, or lower it for tighter CI budgets. + ## How It Works -The supervisor binary (`openshell-sandbox`) is **always side-loaded** from the k3s node filesystem via a read-only `hostPath` volume. It is never baked into sandbox images. This applies to all sandbox pods — whether using the default community base image, a custom image, or a user-built Dockerfile. +The supervisor binary (`openshell-sandbox`) must be delivered by the selected compute driver. The target architecture does not depend on a k3s node hostPath or a cluster image. ```mermaid flowchart TB - subgraph node["K3s Node"] - bin["/opt/openshell/bin/openshell-sandbox - (built into cluster image, updatable via docker cp)"] + subgraph delivery["Supervisor delivery"] + bin["openshell-sandbox + (image, image volume, local binary, or VM rootfs)"] end - node -- "hostPath (readOnly)" --> agent + delivery --> agent subgraph pod["Pod"] subgraph agent["Agent Container"] agent_desc["Image: community base or custom image Command: /opt/openshell/bin/openshell-sandbox - Volume: /opt/openshell/bin (ro hostPath) + Supervisor path configured by compute driver Env: OPENSHELL_SANDBOX_ID, OPENSHELL_ENDPOINT, ... Caps: SYS_ADMIN, NET_ADMIN, SYS_PTRACE"] end end ``` -The server applies these transforms to every sandbox pod template (`sandbox/mod.rs`): +For Kubernetes-backed sandboxes, the driver must ensure every pod template has: -1. Adds a `hostPath` volume named `openshell-supervisor-bin` pointing to `/opt/openshell/bin` on the node. -2. Mounts it read-only at `/opt/openshell/bin` in the agent container. -3. Overrides the agent container's `command` to `/opt/openshell/bin/openshell-sandbox`. -4. Sets `runAsUser: 0` so the supervisor has root privileges for namespace creation, proxy setup, and Landlock/seccomp. +1. A resolvable `openshell-sandbox` entrypoint. +2. Gateway callback environment variables such as `OPENSHELL_SANDBOX_ID`, `OPENSHELL_ENDPOINT`, and `OPENSHELL_SSH_SOCKET_PATH`. +3. TLS and SSH handshake materials when the gateway requires them. +4. The capabilities needed for namespace creation, proxy setup, and Landlock/seccomp. These transforms apply to every generated pod template. @@ -96,8 +99,9 @@ openshell sandbox create --from ./my-sandbox/ # directory with Dockerfile The `openshell-sandbox` supervisor adapts to arbitrary environments: -- **Log file fallback**: Attempts to open `/var/log/openshell.log` for append; silently falls back to stdout-only logging if the path is not writable. +- **Log file fallback**: Attempts to open `/var/log/openshell.log` for append; if the path is not writable, the supervisor keeps console shorthand logging on stderr only. - **Command resolution**: Executes the command from CLI args, then the `OPENSHELL_SANDBOX_COMMAND` env var (set to `sleep infinity` by the server), then `/bin/bash` as a last resort. +- **Startup seccomp prelude**: Before parsing CLI args or starting the async runtime, the supervisor sets `PR_SET_NO_NEW_PRIVS` and installs a narrow seccomp filter that blocks mount/remount, the new mount API syscalls, module loading, kexec, `bpf`, `perf_event_open`, and `userfaultfd`. This closes the privileged remount window while still leaving required child-setup syscalls such as `setns` available. - **Network namespace**: Requires successful namespace creation for proxy isolation; startup fails in proxy mode if required capabilities (`CAP_NET_ADMIN`, `CAP_SYS_ADMIN`) or `iproute2` are unavailable. If the `iptables` package is present, the supervisor installs OUTPUT chain rules (LOG + REJECT) inside the namespace to provide fast-fail behavior (immediate `ECONNREFUSED` instead of a 30-second timeout) and diagnostic logging when processes attempt direct connections that bypass the HTTP CONNECT proxy. If `iptables` is absent, the supervisor logs a warning and continues — core network isolation still works via routing. ## Design Decisions @@ -106,18 +110,19 @@ The `openshell-sandbox` supervisor adapts to arbitrary environments: |----------|-----------| | Unified `--from` flag | Single entry point for community names, Dockerfiles, directories, and image refs — removes the need to know registry paths | | Community name resolution | Bare names like `openclaw` expand to the GHCR community registry, making the common case simple | -| Auto build+push for Dockerfiles | Eliminates the two-step `image push` + `create` workflow for local development | +| Auto build/import for Dockerfiles | Eliminates the two-step build/import + create workflow for local gateway development | | `OPENSHELL_COMMUNITY_REGISTRY` env var | Allows organizations to host their own community sandbox registry | -| hostPath side-load | Supervisor binary lives on the node filesystem — no init container, no emptyDir, no extra image pull. Faster pod startup. | -| Read-only mount in agent | Supervisor binary cannot be tampered with by the workload | +| Driver-owned supervisor delivery | Each compute driver decides how to deliver `openshell-sandbox` for its runtime. | +| Read-only supervisor delivery | The supervisor should be mounted or packaged read-only where the driver supports it, and the startup seccomp prelude blocks remount syscalls that would otherwise reopen it for writes once privileged bootstrap has completed. | | Command override | Ensures `openshell-sandbox` is the entrypoint regardless of the image's default CMD | | Clear `run_as_user/group` for custom images | Prevents startup failure when the image lacks the default `sandbox` user | | Non-fatal log file init | `/var/log/openshell.log` may be unwritable in arbitrary images; falls back to stdout | -| `docker save` / `ctr import` for push | Avoids requiring a registry for local dev; images land directly in the k3s containerd store | +| Local gateway image availability | Dockerfile sources build into the host Docker daemon; managed local gateway deployments import the tag so the selected runtime can resolve it. | | Optional `iptables` for bypass detection | Core network isolation works via routing alone (`iproute2`); `iptables` only adds fast-fail (`ECONNREFUSED`) and diagnostic LOG entries. Making it optional avoids hard failures in minimal images that lack `iptables` while giving better UX when it is available. | ## Limitations - Distroless / `FROM scratch` images are not supported (the supervisor needs glibc and `/proc`) - Missing `iproute2` (or required capabilities) blocks startup in proxy mode because namespace isolation is mandatory -- The supervisor binary must be present on the k3s node at `/opt/openshell/bin/openshell-sandbox` (embedded in the cluster image at build time) +- Local Dockerfile sources are only supported for local gateways; remote gateways require registry image references. +- The selected compute driver must provide an `openshell-sandbox` binary compatible with the sandbox image and host architecture. diff --git a/architecture/sandbox-providers.md b/architecture/sandbox-providers.md index fe5d48a97d..d66b6cd2c5 100644 --- a/architecture/sandbox-providers.md +++ b/architecture/sandbox-providers.md @@ -41,13 +41,67 @@ The gRPC surface is defined in `proto/openshell.proto`: - `CreateProvider` - `GetProvider` - `ListProviders` +- `ListProviderProfiles` +- `GetProviderProfile` - `UpdateProvider` - `DeleteProvider` +## Provider Type Profiles + +Provider type profiles are declarative metadata for provider types. Built-in profiles +live as one YAML document per provider under the top-level `providers/` directory +and are exposed through +`ListProviderProfiles` and `GetProviderProfile`. The profile loader validates the +YAML catalog and materializes the same proto-backed shape that future API imports +will accept. Profiles describe credential names and environment variables, known +network endpoints, expected binaries, category, and whether the provider is +inference-capable. Categories are a proto enum so clients can group and filter +provider types without parsing display strings. Current values are `other`, +`inference`, `agent`, `source_control`, `messaging`, `data`, and `knowledge`. +Agent profiles such as `claude`, `codex`, and `opencode` can still be +inference-capable when their tool talks to an inference API. + +Profiles are additive to provider records. A provider record with only `type`, +`credentials`, and `config` can be matched to built-in profile metadata by +`provider.type`. Profile-generated policy is still opt-in: the gateway composes provider +profile rules only when the gateway-global `providers_v2_enabled` setting is true. + +This keeps the compatibility boundary at the gateway. A gateway without +`providers_v2_enabled=true` keeps the existing credential-only provider behavior, while a +gateway with the flag enabled routes all attached known provider types through the +profile-backed policy path. + +### Provider Policy Composition + +Sandbox policy fetch uses just-in-time composition: + +```text +effective policy = base/static policy + provider profile rules + user rules +``` + +The composed policy is derived data. The sandbox still receives one normal +`SandboxPolicy`, but provider-generated entries are not persisted as user-authored +policy revisions. Full policy replacement and incremental policy updates continue to +mutate the user-authored policy layer. Provider-generated rules are re-added during +composition for each attached provider whose type has a built-in profile. + +Provider-generated network rules use reserved `_provider_*` names derived from the +provider record name. If a user or global policy already has the same key, composition +keeps the policy entry and adds a numeric suffix to the provider entry. Duplicate +host/port endpoints across policy and provider rules are valid; OPA evaluates all +rules, so allow decisions are the union of matching allows and deny rules continue to +win globally. + +Gateway-global policy still overrides sandbox-authored policy. When `providers_v2_enabled` +is true, provider layers compose JIT onto the effective policy source, whether that +source is sandbox-scoped or global. The composed payload is derived data and is not +persisted as a policy revision. + ## Components - `crates/openshell-providers` - canonical provider type normalization and command detection, + - YAML-backed built-in provider profiles, - provider registry and per-provider discovery plugins, - shared discovery engine and context abstraction for testability. - `crates/openshell-cli` @@ -174,6 +228,7 @@ Also supported: - `openshell provider get ` - `openshell provider list` +- `openshell provider list-profiles` - `openshell provider update ...` - `openshell provider delete [...]` @@ -232,6 +287,8 @@ Key behaviors: - Only `credentials` are injected, not `config`. - Invalid env var keys (containing `.`, `-`, spaces, etc.) are skipped. - Credentials are never persisted in the sandbox spec's environment map. +- Provider profiles do not change credential injection in the first iteration. + Injection still uses the existing placeholder environment path. ### Sandbox Supervisor: Fetching Credentials @@ -333,7 +390,7 @@ the full implementation details, encoding rules, and security properties. ### End-to-End Flow -``` +```text CLI: openshell sandbox create -- claude | +-- detect_provider_from_command(["claude"]) -> "claude" diff --git a/architecture/sandbox.md b/architecture/sandbox.md index 096a1bb608..18967cf90f 100644 --- a/architecture/sandbox.md +++ b/architecture/sandbox.md @@ -15,7 +15,8 @@ All paths are relative to `crates/openshell-sandbox/src/`. | `opa.rs` | OPA/Rego policy engine using `regorus` crate -- network evaluation, sandbox config queries, L7 endpoint queries | | `process.rs` | `ProcessHandle` for spawning child processes, privilege dropping, signal handling | | `proxy.rs` | HTTP CONNECT proxy with OPA evaluation, process-identity binding, inference interception, and L7 dispatch | -| `ssh.rs` | Embedded SSH server (`russh` crate) with PTY support and handshake verification | +| `ssh.rs` | Embedded SSH server (`russh` crate) listening on a Unix socket, with PTY support | +| `supervisor_session.rs` | Persistent outbound `ConnectSupervisor` gRPC session to the gateway; bridges `RelayStream` calls to the local SSH daemon's Unix socket | | `identity.rs` | `BinaryIdentityCache` -- SHA256 trust-on-first-use binary integrity | | `procfs.rs` | `/proc` filesystem reading for TCP peer identity resolution and ancestor chain walking | | `grpc_client.rs` | gRPC client for fetching policy, provider environment, inference route bundles, policy polling/status reporting, proposal submission, and log push (`CachedOpenShellClient`) | @@ -28,10 +29,12 @@ All paths are relative to `crates/openshell-sandbox/src/`. | `bypass_monitor.rs` | Background `/dev/kmsg` reader for iptables bypass detection events | | `sandbox/linux/netns.rs` | Network namespace creation, veth pair setup, bypass detection iptables rules, cleanup on drop | | `l7/mod.rs` | L7 types (`L7Protocol`, `TlsMode`, `EnforcementMode`, `L7EndpointConfig`), config parsing, validation, access preset expansion, deprecated `tls` value handling | +| `l7/graphql.rs` | GraphQL-over-HTTP request classifier, body buffering, operation/root-field extraction, and persisted-query metadata handling | | `l7/inference.rs` | Inference API pattern detection (`detect_inference_pattern()`), HTTP request/response parsing and formatting for intercepted inference connections | | `l7/tls.rs` | Ephemeral CA generation (`SandboxCa`), per-hostname leaf cert cache (`CertCache`), TLS termination/connection helpers, `looks_like_tls()` auto-detection | | `l7/relay.rs` | Protocol-aware bidirectional relay with per-request OPA evaluation, credential-injection-only passthrough relay | | `l7/rest.rs` | HTTP/1.1 request/response parsing, body framing (Content-Length, chunked), deny response generation | +| `l7/path.rs` | Request-target canonicalization: percent-decoding, dot-segment resolution, `;params` stripping, encoded-slash policy (opt-in per endpoint via `allow_encoded_slash: true` for upstreams like GitLab that embed `%2F` in paths). Single source of truth for the path both OPA evaluates and the upstream receives. | | `l7/provider.rs` | `L7Provider` trait and `L7Request`/`BodyLength` types | | `secrets.rs` | `SecretResolver` credential placeholder system — placeholder generation, multi-location rewriting (headers, query params, path segments, Basic auth), fail-closed scanning, secret validation, percent-encoding | @@ -64,9 +67,12 @@ flowchart TD L --> L2[Spawn bypass monitor] L2 --> N{SSH enabled?} M --> N - N -- Yes --> O[Spawn SSH server task] - N -- No --> P[Spawn child process] - O --> P + N -- Yes --> O[Spawn SSH server task on Unix socket] + N -- No --> P0{gRPC mode + socket?} + O --> P0 + P0 -- Yes --> P1[Spawn supervisor session task] + P0 -- No --> P[Spawn child process] + P1 --> P P --> Q[Store entrypoint PID] Q --> R{gRPC mode?} R -- Yes --> T[Spawn policy poll task] @@ -94,7 +100,7 @@ flowchart TD - Generate ephemeral CA via `SandboxCa::generate()` using `rcgen` - Write CA cert PEM and combined bundle (system CAs + sandbox CA) to `/etc/openshell-tls/` - Add the TLS directory to `policy.filesystem.read_only` so Landlock allows the child to read it - - Build upstream `ClientConfig` with Mozilla root CAs via `webpki_roots` + - Build upstream `ClientConfig` with Mozilla root CAs (`webpki_roots`) plus system CA certificates from the container's trust store (e.g. corporate CAs added via `update-ca-certificates`) - Create `Arc` wrapping a `CertCache` and the upstream config 6. **Network namespace** (Linux, proxy mode only): @@ -109,18 +115,20 @@ flowchart TD - Build `InferenceContext` via `build_inference_context()` which resolves routes from one of two sources (see [Inference routing context](#inference-routing-context) below) - `ProxyHandle::start_with_bind_addr()` binds a `TcpListener` and spawns an accept loop, passing the inference context to each connection handler -8. **SSH server** (optional): If `--ssh-listen-addr` is provided, spawn an async task running `ssh::run_ssh_server()` with the policy, workdir, netns FD, proxy URL, CA paths, and provider env. +8. **SSH server** (optional): If `--ssh-socket-path` is provided, spawn an async task running `ssh::run_ssh_server()` with the policy, workdir, netns FD, proxy URL, CA paths, and provider env. The value is a filesystem path to the Unix socket the embedded sshd binds. The supervisor waits on a readiness `oneshot` channel before proceeding so that exec requests arriving immediately after pod-ready cannot race against socket bind. -9. **Child process spawning** (`ProcessHandle::spawn()`): - - Build `tokio::process::Command` with inherited stdio and `kill_on_drop(true)` - - Set environment variables: `OPENSHELL_SANDBOX=1`, provider credentials, proxy URLs, TLS trust store paths - - Pre-exec closure (async-signal-safe): `setpgid` (if non-interactive) -> `setns` (enter netns) -> `drop_privileges` -> `sandbox::apply` (Landlock + seccomp) +9. **Supervisor session** (gRPC mode + SSH socket only): If `--sandbox-id`, `--openshell-endpoint`, and an SSH socket path are all set, spawn `supervisor_session::spawn()`. This task opens a persistent outbound bidirectional gRPC stream to the gateway and bridges inbound relay requests to the local SSH daemon. See [Supervisor Session](#supervisor-session) for the full protocol. -10. **Store entrypoint PID**: `entrypoint_pid.store(pid, Ordering::Release)` so the proxy can resolve TCP peer identity via `/proc`. +10. **Child process spawning** (`ProcessHandle::spawn()`): + - Build `tokio::process::Command` with inherited stdio and `kill_on_drop(true)` + - Set environment variables: `OPENSHELL_SANDBOX=1`, provider credentials, proxy URLs, TLS trust store paths + - Pre-exec closure (async-signal-safe): `setpgid` (if non-interactive) -> `setns` (enter netns) -> `drop_privileges` -> `sandbox::apply` (Landlock + seccomp) -11. **Spawn policy poll task** (gRPC mode only): If `sandbox_id`, `openshell_endpoint`, and an OPA engine are all present, spawn `run_policy_poll_loop()` as a background tokio task. This task polls the gateway for policy updates and hot-reloads the OPA engine when a new version is detected. See [Policy Reload Lifecycle](#policy-reload-lifecycle) for details. +11. **Store entrypoint PID**: `entrypoint_pid.store(pid, Ordering::Release)` so the proxy can resolve TCP peer identity via `/proc`. -12. **Wait with timeout**: If `--timeout > 0`, wrap `handle.wait()` in `tokio::time::timeout()`. On timeout, kill the process and return exit code 124. +12. **Spawn policy poll task** (gRPC mode only): If `sandbox_id`, `openshell_endpoint`, and an OPA engine are all present, spawn `run_policy_poll_loop()` as a background tokio task. This task polls the gateway for policy updates and hot-reloads the OPA engine when a new version is detected. See [Policy Reload Lifecycle](#policy-reload-lifecycle) for details. + +13. **Wait with timeout**: If `--timeout > 0`, wrap `handle.wait()` in `tokio::time::timeout()`. On timeout, kill the process and return exit code 124. ## Policy Model @@ -237,6 +245,7 @@ Two evaluation methods exist: `evaluate_network()` for the legacy bool-based pat #### `evaluate_network(input: &NetworkInput) -> Result` Input JSON shape: + ```json { "exec": { @@ -252,6 +261,7 @@ Input JSON shape: ``` Evaluates three Rego rules: + 1. `data.openshell.sandbox.allow_network` -> bool 2. `data.openshell.sandbox.deny_reason` -> string 3. `data.openshell.sandbox.matched_network_policy` -> string (or `Undefined`) @@ -266,6 +276,7 @@ Uses the same input JSON shape as `evaluate_network()`. Evaluates the `data.open - `"deny"` -- network connections not allowed by policy The Rego logic: + 1. If `network_policy_for_request` exists (endpoint + binary match), return `"allow"` 2. Default: `"deny"` @@ -282,18 +293,22 @@ The proxy calls `evaluate_network_action()` (not `evaluate_network()`) as its ma ### L7 endpoint config query -After L4 allows a connection, `query_endpoint_config(input)` evaluates `data.openshell.sandbox.matched_endpoint_config` to get the full endpoint object. If the endpoint has a `protocol` field, `l7::parse_l7_config()` extracts the L7 config for protocol-aware inspection. +After L4 allows a connection, `query_endpoint_config_with_generation(input)` evaluates `data.openshell.sandbox.matched_endpoint_config` to get the full endpoint object and the policy generation used for the query. If the endpoint has a `protocol` field, `l7::parse_l7_config()` extracts the L7 config for protocol-aware inspection. ### Engine cloning for L7 -`clone_engine_for_tunnel()` clones the inner `regorus::Engine`. With the `arc` feature, this shares compiled policy via `Arc` and only duplicates interpreter state (microseconds). The cloned engine is wrapped in its own `std::sync::Mutex` and used by the L7 relay without contention on the main engine. +`clone_engine_for_tunnel(expected_generation)` clones the inner `regorus::Engine` only if the current policy generation still matches the endpoint config generation captured above. With the `arc` feature, this shares compiled policy via `Arc` and only duplicates interpreter state (microseconds). The cloned engine is wrapped in a generation-bound `TunnelPolicyEngine` and used by the L7 relay without contention on the main engine. + +The L7 relay checks the captured generation before parsing, evaluating, and forwarding each request. If a policy reload has advanced the shared generation, the relay closes the tunnel before forwarding more bytes. This applies live policy changes to the next L7 request on a keep-alive tunnel and avoids pairing stale endpoint config with a newer policy engine. HTTP passthrough tunnels without endpoint `protocol` are also generation-bound so credential-injection-only keep-alive tunnels close after a reload before forwarding another request. + +Raw streams are connection-scoped and outside the L7 live-reload guarantee. This includes `tls: skip`, binary/non-HTTP CONNECT tunnels, SQL audit fallback passthrough, HTTP upgrades such as WebSocket after `101 Switching Protocols`, and already-forwarded long-lived response bodies such as SSE. Policy reloads affect the next connection or the next parsed HTTP request; they do not interrupt raw byte streams that have already moved outside the request parser. ### Hot reload Two reload methods exist: - **`reload(policy, data_yaml)`**: Builds a new engine from raw Rego + YAML strings and atomically replaces the inner engine. Used in tests and by the file-mode path. -- **`reload_from_proto(proto)`**: Builds a new engine through the same validated pipeline as `from_proto()` -- proto-to-JSON conversion, L7 validation, access preset expansion -- then atomically swaps the inner `regorus::Engine`. On success, all subsequent `evaluate_network_action()` and `query_endpoint_config()` calls use the new policy. On failure (e.g., L7 validation errors), the previous engine is untouched (last-known-good behavior). This is the method used by the policy poll loop for live reloads in gRPC mode. +- **`reload_from_proto(proto)`**: Builds a new engine through the same validated pipeline as `from_proto()` -- proto-to-JSON conversion, L7 validation, access preset expansion -- then atomically swaps the inner `regorus::Engine`. On success, all subsequent `evaluate_network_action()` and `query_endpoint_config()` calls use the new policy, and the engine generation increments so active L7 tunnels close before forwarding another request under stale state. On failure (e.g., L7 validation errors), the previous engine and generation are untouched (last-known-good behavior). This is the method used by the policy poll loop for live reloads in gRPC mode. Both methods hold the `Mutex` only for the final swap (`*engine = new_engine`), so evaluation is blocked for only the duration of a pointer-sized assignment. @@ -387,6 +402,7 @@ pub struct SettingsPollResult { ``` Methods: + - **`connect(endpoint)`**: Establish an mTLS channel and return a new client. - **`poll_settings(sandbox_id)`**: Call `GetSandboxSettings` RPC and return a `SettingsPollResult` containing policy payload (optional), policy metadata, effective config revision, policy source, global policy version, and the effective settings map (for diff logging). - **`report_policy_status(sandbox_id, version, loaded, error_msg)`**: Call `ReportPolicyStatus` RPC with the appropriate `PolicyStatus` enum value (`Loaded` or `Failed`). @@ -397,6 +413,7 @@ Methods: The gateway assigns a monotonically increasing version number to each sandbox policy revision. `GetSandboxSettingsResponse` carries the full effective configuration: policy payload, effective settings map (with per-key scope indicators), a `config_revision` fingerprint that changes when any effective input changes (policy, settings, or source), and a `policy_source` field indicating whether the policy came from the sandbox's own history or from a global override. Proto messages involved: + - `GetSandboxSettingsResponse` (`proto/sandbox.proto`): `policy`, `version`, `policy_hash`, `settings` (map of `EffectiveSetting`), `config_revision`, `policy_source`, `global_policy_version` - `EffectiveSetting` (`proto/sandbox.proto`): `SettingValue value`, `SettingScope scope` - `SettingScope` enum: `UNSPECIFIED`, `SANDBOX`, `GLOBAL` @@ -442,10 +459,21 @@ Landlock restricts the child process's filesystem access to an explicit allowlis 7. Call `ruleset.restrict_self()` -- this applies to the calling process and all descendants Kernel-level error behavior (e.g., Landlock ABI unavailable) depends on `LandlockCompatibility`: + - `BestEffort`: Log a warning and continue without filesystem isolation - `HardRequirement`: Return a fatal error, aborting the sandbox -**Baseline path filtering**: System-injected baseline paths (e.g., `/app`) are pre-filtered by `enrich_proto_baseline_paths()` / `enrich_sandbox_baseline_paths()` using `Path::exists()` before they reach Landlock. User-specified paths are not pre-filtered -- they are evaluated at Landlock apply time so misconfigurations surface as warnings or errors. +**Baseline path filtering**: System-injected baseline paths (e.g., `/app`) are pre-filtered by `enrich_proto_baseline_paths()` / `enrich_sandbox_baseline_paths()` using `Path::exists()` before they reach Landlock. If a baseline `read_write` path is already present in `read_only`, enrichment skips the promotion so explicit policy intent is preserved. User-specified paths are not pre-filtered -- they are evaluated at Landlock apply time so misconfigurations surface as warnings or errors. + +**GPU baseline paths**: The supervisor currently infers GPU baseline paths from +device nodes and NVIDIA runtime paths visible inside the sandbox container. The +Docker compute driver can request CDI GPU injection, but this implementation +does not pass CDI metadata into the supervisor. Future device-specific CDI +selection may need follow-up work so the supervisor can enrich Landlock using +the requested CDI device's actual device nodes and mounted library paths. That +design must work for remote Docker daemons, where Docker-reported CDI spec +directories are paths on the daemon host and may not be readable by the gateway +process or the sandbox supervisor. ### Seccomp syscall filtering @@ -456,6 +484,7 @@ Seccomp provides three layers of syscall restriction: socket domain blocks, unco **Skipped entirely** in `Allow` mode. Setup: + 1. `prctl(PR_SET_NO_NEW_PRIVS, 1)` -- required before seccomp 2. `seccompiler::apply_filter()` with default action `Allow` and per-rule action `Errno(EPERM)` @@ -505,7 +534,7 @@ The network namespace creates an isolated network stack where the sandboxed proc #### Topology -``` +```text HOST NAMESPACE SANDBOX NAMESPACE ----------------- ----------------- veth-h-{uuid} veth-s-{uuid} @@ -533,6 +562,7 @@ Each step has rollback on failure -- if any `ip` command fails, previously creat #### Cleanup on drop `NetworkNamespace` implements `Drop`: + 1. Close the namespace FD 2. Delete the host-side veth (`ip link delete veth-h-{id}`) -- this automatically removes the peer 3. Delete the namespace (`ip netns delete sandbox-{id}`) @@ -570,7 +600,7 @@ The proxy port defaults to `3128` unless the policy specifies a different `http_ 1. Opens `/dev/kmsg` in read mode and seeks to end (skips historical messages) 2. Reads lines via `BufReader`, filtering for the namespace-specific prefix `openshell:bypass:{namespace_name}:` 3. Parses iptables LOG format via `parse_kmsg_line()`, extracting `DST`, `DPT`, `SPT`, `PROTO`, and `UID` fields -4. Resolves process identity for TCP events via `procfs::resolve_tcp_peer_identity()` (best-effort — requires a valid entrypoint PID and non-zero source port) +4. Resolves process identity for TCP events via multi-owner socket inode lookup (best-effort — requires a valid entrypoint PID and non-zero source port). If multiple processes hold the same socket with different executable identities, the event is marked ambiguous instead of attributing it to one PID. 5. Emits a structured `tracing::warn!()` event with the tag `BYPASS_DETECT` 6. Sends a `DenialEvent` to the denial aggregator channel (if available) @@ -801,7 +831,7 @@ Every CONNECT request to a non-`inference.local` target produces an `info!()` lo ### SSRF protection (internal IP rejection) -After OPA allows a connection, the proxy resolves DNS and rejects any host that resolves to an internal IP address (loopback, RFC 1918 private, link-local, or IPv4-mapped IPv6 equivalents). This defense-in-depth measure prevents SSRF attacks where an allowed hostname is pointed at internal infrastructure. The check is implemented by `resolve_and_reject_internal()` which calls `tokio::net::lookup_host()` and validates every resolved address via `is_internal_ip()`. If any resolved IP is internal, the connection receives a `403 Forbidden` response and a warning is logged. See [SSRF Protection](security-policy.md#ssrf-protection-internal-ip-rejection) for the full list of blocked ranges. +After OPA allows a connection, the proxy resolves the host using the sandbox's `/etc/hosts` first on Linux (via `/proc//root/etc/hosts`, which picks up Kubernetes `hostAliases`), then falls back to DNS. It rejects any host that resolves to an internal IP address (loopback, RFC 1918 private, link-local, or IPv4-mapped IPv6 equivalents). This defense-in-depth measure prevents SSRF attacks where an allowed hostname is pointed at internal infrastructure. The check is implemented by `resolve_and_reject_internal()`, which validates every resolved address via `is_internal_ip()`. If any resolved IP is internal, the connection receives a `403 Forbidden` response and a warning is logged. `hostAliases` only affect name resolution — private destinations still need `allowed_ips`. See [SSRF Protection](security-policy.md#ssrf-protection-internal-ip-rejection) for the full list of blocked ranges. IP classification helpers (`is_always_blocked_ip`, `is_always_blocked_net`, `is_internal_ip`) are shared from `openshell_core::net`. The `parse_allowed_ips` function rejects entries overlapping always-blocked ranges (loopback, link-local, unspecified) at load time with a hard error, and `implicit_allowed_ips_for_ip_host` skips synthesis for always-blocked literal IP hosts. The mechanistic mapper filters proposals for always-blocked destinations to prevent infinite TUI notification loops. @@ -839,9 +869,9 @@ The interception steps: Pattern matching strips query strings. Exact path comparison is used for most patterns; the `/v1/models/*` pattern matches `/v1/models` itself or any path under `/v1/models/` (e.g., `/v1/models/gpt-4.1`). -4. **Header sanitization**: For matched inference requests, the proxy strips credential headers (`Authorization`, `x-api-key`) and framing/hop-by-hop headers (`host`, `content-length`, `transfer-encoding`, `connection`, etc.). The router rebuilds correct framing for the forwarded body. +4. **Header sanitization**: For matched inference requests, the proxy passes the parsed headers to the router. The router applies a route-aware allowlist before forwarding: common inference headers (`content-type`, `accept`, `accept-encoding`, `user-agent`), provider-specific passthrough headers (for example `openai-organization`, `x-model-id`, `anthropic-version`, `anthropic-beta`), and any route default header names. It always strips client-supplied credential headers (`Authorization`, `x-api-key`) and framing/hop-by-hop headers (`host`, `content-length`, `transfer-encoding`, `connection`, etc.). The router rebuilds correct framing for the forwarded body. -5. **Local routing**: Matched requests are executed by calling `Router::proxy_with_candidates_streaming()`, passing the detected protocol, HTTP method, path, sanitized headers, body, and the cached `ResolvedRoute` list from `InferenceContext`. The router selects the first route whose `protocols` list contains the source protocol (see [Inference Routing -- Response streaming](inference-routing.md#response-streaming) for details). When forwarding to the backend, the router rewrites the request: the route's `api_key` replaces the `Authorization` header, the `Host` header is set to the backend endpoint, and the `"model"` field in the JSON request body is replaced with the route's configured `model` value. If the request body is not valid JSON or does not contain a `"model"` key, the body is forwarded unchanged. +5. **Local routing**: Matched requests are executed by calling `Router::proxy_with_candidates_streaming()`, passing the detected protocol, HTTP method, path, original parsed headers, body, and the cached `ResolvedRoute` list from `InferenceContext`. The router selects the first route whose `protocols` list contains the source protocol (see [Inference Routing -- Response streaming](inference-routing.md#response-streaming) for details). When forwarding to the backend, the router rewrites the request: the route's `api_key` replaces the client auth header, the `Host` header is set to the backend endpoint, only allowlisted request headers survive, and the `"model"` field in the JSON request body is replaced with the route's configured `model` value. If the request body is not valid JSON or does not contain a `"model"` key, the body is forwarded unchanged. 6. **Response handling (streaming)**: - On success: response headers are sent back to the client immediately as an HTTP/1.1 response with `Transfer-Encoding: chunked`, using `format_http_response_header()`. Framing/hop-by-hop headers are stripped from the upstream response. Body chunks are then forwarded incrementally as they arrive from the backend via `StreamingProxyResponse::next_chunk()`, each wrapped in HTTP chunked encoding by `format_chunk()`. The stream is terminated with a `0\r\n\r\n` chunk terminator. This ensures time-to-first-byte reflects the backend's first token latency rather than the full generation time. @@ -883,28 +913,29 @@ pub struct InferenceContext { #### Design decision: standalone capability -The sandbox is designed to operate both as part of a cluster and as a standalone component without any cluster infrastructure. This is intentional -- it enables local development workflows (e.g., a developer running a sandbox against a local LLM server without deploying the full stack), CI/CD environments where sandboxes run as isolated test harnesses, and air-gapped deployments where the gateway is not available. Everything the sandbox needs -- policy, inference routes -- can be provided without any dependency on the control plane. +The sandbox is designed to operate both under a gateway-managed compute platform and as a standalone component without gateway infrastructure. This is intentional -- it enables local development workflows (e.g., a developer running a sandbox against a local LLM server without deploying the full stack), CI/CD environments where sandboxes run as isolated test harnesses, and air-gapped deployments where the gateway is not available. Everything the sandbox needs -- policy, inference routes -- can be provided without any dependency on the control plane. #### Route sources (priority order) -1. **Route file (standalone mode)**: `--inference-routes` / `OPENSHELL_INFERENCE_ROUTES` points to a YAML file parsed by `RouterConfig::load_from_file()`. Routes are resolved via `config.resolve_routes()`. File loading or parsing errors are fatal (fail-fast), but an empty route list gracefully disables inference routing (returns `None`). The route file always takes precedence -- if both a route file and cluster credentials are present, the route file wins and the cluster bundle is not fetched. +1. **Route file (standalone mode)**: `--inference-routes` / `OPENSHELL_INFERENCE_ROUTES` points to a YAML file parsed by `RouterConfig::load_from_file()`. Routes are resolved via `config.resolve_routes()`. File loading or parsing errors are fatal (fail-fast), but an empty route list gracefully disables inference routing (returns `None`). The route file always takes precedence -- if both a route file and gateway credentials are present, the route file wins and the gateway bundle is not fetched. + +2. **Gateway bundle (gateway mode)**: When `openshell_endpoint` is available (and no route file is configured), routes are fetched from the gateway via `grpc_client::fetch_inference_bundle()`, which calls the `GetInferenceBundle` gRPC RPC on the `Inference` service. The RPC takes no arguments (the bundle is gateway-scoped, not per-sandbox). The gateway returns a `GetInferenceBundleResponse` containing resolved `ResolvedRoute` entries for the managed gateway route. These proto messages are converted to router `ResolvedRoute` structs by `bundle_to_resolved_routes()`, which maps provider types to auth headers and default headers via `openshell_core::inference::auth_for_provider_type()`. -2. **Cluster bundle (cluster mode)**: When `openshell_endpoint` is available (and no route file is configured), routes are fetched from the gateway via `grpc_client::fetch_inference_bundle()`, which calls the `GetInferenceBundle` gRPC RPC on the `Inference` service. The RPC takes no arguments (the bundle is cluster-scoped, not per-sandbox). The gateway returns a `GetInferenceBundleResponse` containing resolved `ResolvedRoute` entries for the managed cluster route. These proto messages are converted to router `ResolvedRoute` structs by `bundle_to_resolved_routes()`, which maps provider types to auth headers and default headers via `openshell_core::inference::auth_for_provider_type()`. +3. **No source**: If neither route file nor gateway credentials are configured, `build_inference_context()` returns `None` and inference routing is disabled. -3. **No source**: If neither route file nor cluster credentials are configured, `build_inference_context()` returns `None` and inference routing is disabled. +#### Gateway mode graceful degradation -#### Cluster mode graceful degradation +In gateway mode, `fetch_inference_bundle()` failures are handled based on the error type: -In cluster mode, `fetch_inference_bundle()` failures are handled based on the error type: - gRPC `PermissionDenied` or `NotFound` (detected via error message string matching): sandbox has no inference policy -- inference routing is silently disabled. - Other errors: logged as a warning, inference routing is disabled. - Empty initial route bundle: inference routing stays enabled with an empty cache and background refresh continues. -Route sources handle empty route lists differently: file mode disables inference routing when the file resolves to zero routes, while cluster mode keeps inference routing active with an empty cache so refresh can pick up routes created later. File *loading errors* (missing file, parse failure) are fatal, while cluster *fetch errors* are non-fatal. +Route sources handle empty route lists differently: file mode disables inference routing when the file resolves to zero routes, while gateway mode keeps inference routing active with an empty cache so refresh can pick up routes created later. File *loading errors* (missing file, parse failure) are fatal, while gateway *fetch errors* are non-fatal. #### Background route cache refresh -In cluster mode (when no route file is configured), `spawn_route_refresh()` starts a background tokio task that refreshes the route cache every 30 seconds (`ROUTE_REFRESH_INTERVAL_SECS`). The task calls `fetch_inference_bundle()` on each tick and replaces the `RwLock>` contents. On fetch failure, the task logs a warning and keeps the stale routes. The `MissedTickBehavior::Skip` policy prevents refresh storms after temporary gateway outages. +In gateway mode (when no route file is configured), `spawn_route_refresh()` starts a background tokio task that refreshes the route cache every 30 seconds (`ROUTE_REFRESH_INTERVAL_SECS`). The task calls `fetch_inference_bundle()` on each tick and replaces the `RwLock>` contents. On fetch failure, the task logs a warning and keeps the stale routes. The `MissedTickBehavior::Skip` policy prevents refresh storms after temporary gateway outages. ```mermaid flowchart TD @@ -924,7 +955,7 @@ flowchart TD M -- Yes --> L M -- No --> N[Warn + None] H -- No --> L - F --> O[spawn_route_refresh if cluster mode] + F --> O[spawn_route_refresh if gateway mode] G --> O ``` @@ -936,7 +967,7 @@ flowchart TD After a CONNECT is allowed, the SSRF check passes, and the upstream TCP connection is established, the proxy determines how to handle the tunnel traffic. TLS detection is automatic — the proxy peeks the first bytes of the client stream to decide. -1. **Query L7 config**: `query_l7_config()` asks the OPA engine for `matched_endpoint_config`. If the endpoint has a `protocol` field, parse it into `L7EndpointConfig`. +1. **Query L7 route**: `query_l7_route_snapshot()` asks the OPA engine for `matched_endpoint_config` and the current policy generation. If the endpoint has a `protocol` field, parse it into a generation-bound `L7ConfigSnapshot`. If the endpoint has no `protocol`, retain the generation for HTTP passthrough keep-alive tunnels. 2. **Check for `tls: skip`**: If the endpoint has `tls: skip`, bypass all auto-detection and relay raw bytes via `copy_bidirectional()`. This is the escape hatch for client-cert mTLS or non-standard protocols. @@ -944,15 +975,15 @@ After a CONNECT is allowed, the SSRF check passes, and the upstream TCP connecti 4. **TLS detected** (`is_tls = true`): - Terminate TLS unconditionally via `tls_terminate_client()` + `tls_connect_upstream()`. This happens for all HTTPS endpoints, not just those with L7 config. - - If L7 config is present: clone the OPA engine (`clone_engine_for_tunnel()`), run `relay_with_inspection()` for per-request policy evaluation. - - If no L7 config: run `relay_passthrough_with_credentials()` — parses HTTP minimally to inject credentials (via `SecretResolver`) and log requests, but does not evaluate L7 OPA rules. This enables credential injection on all HTTPS endpoints without requiring `protocol` in the policy. + - If L7 config is present: clone the OPA engine for the captured generation (`clone_engine_for_tunnel(generation)`), run `relay_with_inspection()` for per-request policy evaluation. If the generation changed between config lookup and clone, close the tunnel before inspection. + - If no L7 config: run `relay_passthrough_with_credentials()` — parses HTTP minimally to inject credentials (via `SecretResolver`) and log requests, but does not evaluate L7 OPA rules. The passthrough relay is bound to the policy generation captured at connection setup and closes before forwarding another request after a reload. This enables credential injection on all HTTPS endpoints without requiring `protocol` in the policy. - If TLS state is not configured: fall back to raw `copy_bidirectional()` with a warning. 5. **Plaintext HTTP detected** (`is_http = true`, `is_tls = false`): - - If L7 config present: clone OPA engine, run `relay_with_inspection()` directly on the plaintext streams. - - If no L7 config: run `relay_passthrough_with_credentials()` for credential injection and observability. + - If L7 config present: clone the OPA engine for the captured generation, run `relay_with_inspection()` directly on the plaintext streams. + - If no L7 config: run `relay_passthrough_with_credentials()` for credential injection and observability, with the same per-request generation guard. -6. **Neither TLS nor HTTP**: Raw `copy_bidirectional()` tunnel (binary protocols, SSH-over-CONNECT, etc.). +6. **Neither TLS nor HTTP**: Raw `copy_bidirectional()` tunnel (binary protocols, SSH-over-CONNECT, etc.). These raw streams are connection-scoped and continue until either side closes; live policy reload does not interrupt them. ```mermaid flowchart TD @@ -976,7 +1007,7 @@ flowchart TD **Files:** `crates/openshell-sandbox/src/l7/` -The L7 subsystem inspects application-layer traffic within CONNECT tunnels. Instead of raw `copy_bidirectional`, each request is parsed, evaluated against OPA rules, and either forwarded or blocked. +The L7 subsystem inspects application-layer traffic within CONNECT tunnels. Instead of raw `copy_bidirectional`, each request is parsed, evaluated against OPA rules, and either forwarded or blocked. The relay uses a generation-bound policy snapshot; after a successful policy reload, an existing L7 keep-alive tunnel closes before forwarding another request. Once an HTTP request has upgraded into a raw stream, or when a response body is a long-lived stream, that stream is connection-scoped and is not interrupted by L7 live reload. ### Architecture @@ -998,12 +1029,12 @@ flowchart LR | Type | Definition | Purpose | |------|-----------|---------| -| `L7Protocol` | `Rest`, `Sql` | Supported application protocols | +| `L7Protocol` | `Rest`, `Graphql`, `Sql` | Supported application protocols | | `TlsMode` | `Auto` (default), `Skip` | TLS handling strategy — `Auto` peeks first bytes and terminates if TLS is detected; `Skip` bypasses detection entirely | | `EnforcementMode` | `Audit`, `Enforce` | What to do on L7 deny (log-only vs block) | -| `L7EndpointConfig` | `{ protocol, tls, enforcement }` | Per-endpoint L7 configuration | +| `L7EndpointConfig` | `{ protocol, path, tls, enforcement, allow_encoded_slash, graphql_max_body_bytes }` | Per-endpoint L7 configuration, including optional path scoping for shared host:port APIs | | `L7Decision` | `{ allowed, reason, matched_rule }` | Result of L7 evaluation | -| `L7RequestInfo` | `{ action, target, query_params }` | HTTP method, path, and decoded query multimap for policy evaluation | +| `L7RequestInfo` | `{ action, target, query_params, graphql }` | HTTP method, path, decoded query multimap, and optional GraphQL classification for policy evaluation | ### Access presets @@ -1011,9 +1042,9 @@ Policy data supports shorthand `access` presets that expand into explicit `rules | Preset | Expands to | |--------|-----------| -| `read-only` | `GET **`, `HEAD **`, `OPTIONS **` | -| `read-write` | `GET **`, `HEAD **`, `OPTIONS **`, `POST **`, `PUT **`, `PATCH **` | -| `full` | `* **` (all methods, all paths) | +| `read-only` | REST: `GET **`, `HEAD **`, `OPTIONS **`; GraphQL: `query` | +| `read-write` | REST: `GET **`, `HEAD **`, `OPTIONS **`, `POST **`, `PUT **`, `PATCH **`; GraphQL: `query`, `mutation` | +| `full` | REST: `* **`; GraphQL: `operation_type: "*"` | Expansion happens in `expand_access_presets()` before the Rego engine loads the data. The `rules` and `access` fields are mutually exclusive (validated at startup). @@ -1022,15 +1053,20 @@ Expansion happens in `expand_access_presets()` before the Rego engine loads the `validate_l7_policies()` runs at engine load time and returns `(errors, warnings)`: **Errors** (block startup): + - `rules` and `access` both specified on same endpoint - `protocol` specified without `rules` or `access` +- unknown `protocol` - `protocol: sql` with `enforcement: enforce` (SQL parsing not available in v1) - Empty `rules` array (would deny all traffic) +- invalid GraphQL operation types, persisted-query mode, body limit, or rule shape **Warnings** (logged): + - `tls: terminate` or `tls: passthrough` on any endpoint (deprecated — TLS termination is now automatic; use `tls: skip` to disable) - `tls: skip` with L7 rules on port 443 (L7 inspection cannot work on encrypted traffic) - Unknown HTTP method in rules +- GraphQL-specific fields on non-GraphQL endpoints ### TLS termination (auto-detect) @@ -1039,25 +1075,29 @@ Expansion happens in `expand_access_presets()` before the Rego engine loads the TLS termination is automatic. The proxy peeks the first bytes of every CONNECT tunnel and terminates TLS whenever a ClientHello is detected. This enables credential injection and L7 inspection on all HTTPS endpoints without requiring explicit `tls: terminate` in the policy. The `tls` field defaults to `Auto`; use `tls: skip` to opt out entirely (e.g., for client-cert mTLS to upstream). **Ephemeral CA lifecycle:** + 1. At sandbox startup, `SandboxCa::generate()` creates a self-signed CA (CN: "OpenShell Sandbox CA") using `rcgen` 2. The CA cert PEM and a combined bundle (system CAs + sandbox CA) are written to `/etc/openshell-tls/` 3. The sandbox CA cert path is set as `NODE_EXTRA_CA_CERTS` (additive for Node.js) 4. The combined bundle is set as `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE` (replaces defaults for OpenSSL, Python requests, curl) **TLS auto-detection** (`looks_like_tls()`): + - Peeks up to 8 bytes from the client stream - Checks for TLS ClientHello pattern: byte 0 = `0x16` (ContentType::Handshake), byte 1 = `0x03` (TLS major version), byte 2 ≤ `0x04` (minor version, covering SSL 3.0 through TLS 1.3) - Returns `false` for plaintext HTTP, SSH, or other binary protocols **Per-hostname leaf cert generation:** + - `CertCache` maps hostnames to `CertifiedLeaf` structs (cert chain + private key) - First request for a hostname generates a leaf cert signed by the sandbox CA via `rcgen` - Cache has a hard limit of 256 entries; on overflow, the entire cache is cleared (sufficient for sandbox scale) - Each leaf cert chain contains two certs: the leaf and the CA **Connection flow (when TLS is detected):** + 1. `tls_terminate_client()`: Accept TLS from the sandboxed client using a `ServerConfig` with the hostname-specific leaf cert. ALPN: `http/1.1`. -2. `tls_connect_upstream()`: Connect TLS to the real upstream using a `ClientConfig` with Mozilla root CAs (`webpki_roots`). ALPN: `http/1.1`. +2. `tls_connect_upstream()`: Connect TLS to the real upstream using a `ClientConfig` with Mozilla root CAs (`webpki_roots`) and system CA certificates. ALPN: `http/1.1`. 3. Proxy now holds plaintext on both sides. If L7 config is present, runs `relay_with_inspection()`. Otherwise, runs `relay_passthrough_with_credentials()` for credential injection without L7 evaluation. System CA bundles are searched at well-known paths: `/etc/ssl/certs/ca-certificates.crt` (Debian/Ubuntu), `/etc/pki/tls/certs/ca-bundle.crt` (RHEL), `/etc/ssl/ca-bundle.pem` (openSUSE), `/etc/ssl/cert.pem` (Alpine/macOS). @@ -1201,18 +1241,28 @@ Implements `L7Provider` for HTTP/1.1: - **`looks_like_http()`**: Protocol detection via first-byte peek -- checks for standard HTTP method prefixes (GET, HEAD, POST, PUT, DELETE, PATCH, OPTIONS, CONNECT, TRACE). +### GraphQL protocol classifier + +**File:** `crates/openshell-sandbox/src/l7/graphql.rs` + +GraphQL inspection reuses the HTTP parser, then buffers the request body up to `graphql_max_body_bytes` for classification. It supports `GET` and `POST` GraphQL-over-HTTP envelopes, JSON batches, named operations, root fragment expansion, Apollo persisted-query hashes, and saved-query IDs (`id`, `documentId`, `queryId`). The classifier emits `GraphqlRequestInfo` with operation type, optional operation name, root fields, and persisted-query identifiers. + +Hash-only or saved-query-only requests cannot be parsed into operation fields. They are denied unless the endpoint sets `persisted_queries: allow_registered` and provides a trusted `graphql_persisted_queries` entry for the hash or ID. Batch requests are fail-closed: any malformed, denied, or unregistered operation denies the whole HTTP request. + ### Per-request L7 evaluation `relay_with_inspection()` in `crates/openshell-sandbox/src/l7/relay.rs` is the main relay loop: -1. Parse one HTTP request from client via the provider +1. Parse one HTTP request from client via the provider. Parser and path-canonicalization failures close the connection and emit a denied OCSF network event with the rejection reason in `status_detail`. 2. Resolve credential placeholders in the request target via `rewrite_target_for_eval()`. OPA receives the redacted path (`[CREDENTIAL]` markers); the resolved path goes only to upstream. If resolution fails, return HTTP 500 and close the connection. -3. Build L7 input JSON with `request.method`, the **redacted** `request.path`, `request.query_params`, plus the CONNECT-level context (host, port, binary, ancestors, cmdline) +3. Build L7 input JSON with `request.method`, the **redacted** `request.path`, `request.query_params`, optional `request.graphql`, plus the CONNECT-level context (host, port, binary, ancestors, cmdline) 4. Evaluate `data.openshell.sandbox.allow_request` and `data.openshell.sandbox.request_deny_reason` 5. Log the L7 decision (tagged `L7_REQUEST`) using the redacted target — real credential values never appear in logs 6. If allowed (or audit mode): relay request to upstream via `relay_http_request_with_resolver()` (which rewrites all remaining credential placeholders in headers, query parameters, path segments, and Basic auth tokens) and relay the response back to client, then loop 7. If denied in enforce mode: send 403 (using redacted target in the response body) and close the connection +Before parsing, before evaluation, and before forwarding each request, the relay checks whether its captured policy generation still matches the shared engine generation. If not, it emits a denied OCSF network event and closes the tunnel without forwarding the request. + ## Process Identity ### SHA256 TOFU (Trust-On-First-Use) @@ -1226,11 +1276,13 @@ each cached entry stores: - File fingerprint (`len`, `mtime`, `ctime`, and on Unix `dev` + `inode`) `verify_or_cache(path)`: + - **First call for a path**: Compute SHA256 via `procfs::file_sha256()`, store as the "golden" hash plus fingerprint, return the hash. - **Subsequent calls, unchanged fingerprint**: Return cached hash without re-hashing the file. - **Subsequent calls, changed fingerprint**: Recompute SHA256 and compare with cached value. Return `Ok(hash)` on match; return `Err` on mismatch (binary tampered/replaced mid-sandbox). The TOFU model means: + - No hashes are specified in policy data -- the first observed binary is trusted - Once trusted, the binary cannot change for the sandbox's lifetime - Both the immediate binary and all ancestor binaries are TOFU-verified @@ -1241,22 +1293,26 @@ The TOFU model means: The proxy resolves which binary is making each network request by inspecting `/proc`. -**`resolve_tcp_peer_identity(entrypoint_pid, peer_port) -> (PathBuf, u32)`** +**`resolve_tcp_peer_socket_owners(entrypoint_pid, peer_port) -> TcpPeerSocketOwners`** ```mermaid flowchart TD A["Parse /proc/{entrypoint}/net/tcp + tcp6"] --> B[Find ESTABLISHED socket with matching local port] B --> C[Extract socket inode] - C --> D["BFS collect descendants of entrypoint via /proc/{pid}/task/{tid}/children"] - D --> E["Scan /proc/{pid}/fd/* for socket:[inode] symlink"] - E --> F{Found?} - F -- Yes --> G["Read /proc/{pid}/exe -> binary path"] - F -- No --> H["Fallback: scan all /proc PIDs"] - H --> G + C --> D["BFS collect descendants of entrypoint via /proc/{pid}/task/{tid}/children, deduping PIDs"] + D --> E["Scan every descendant /proc/{pid}/fd/* for socket:[inode] symlink"] + E --> F["Fallback: scan all /proc PIDs not already checked"] + F --> G["Return all socket owner PIDs with source/depth metadata"] + G --> H["Read /proc/{pid}/exe, TOFU-check each owner and ancestor"] + H --> I{All owners same policy identity?} + I -- Yes --> J["Evaluate OPA once with the shared identity"] + I -- No --> K["Deny as ambiguous shared socket ownership"] ``` Both IPv4 (`/proc/{pid}/net/tcp`) and IPv6 (`/proc/{pid}/net/tcp6`) tables are checked because some clients (notably gRPC C-core) use `AF_INET6` sockets with IPv4-mapped addresses. +Multiple processes can hold the same socket inode after `fork()` or fd inheritance across `execve()`. The proxy treats that as ambiguous unless all socket owners resolve to the same policy identity: binary path, TOFU hash, ancestor chain, and cmdline-derived absolute paths. Ambiguous ownership is denied before OPA evaluation so a trusted co-owner cannot accidentally authorize traffic for a different process. + **`collect_ancestor_binaries(pid, stop_pid) -> Vec`**: Walk the PPid chain via `/proc/{pid}/status`, collecting `binary_path()` for each ancestor. Stops at PID 1, `stop_pid` (entrypoint), or after 64 levels (safety limit). Does not include `pid` itself. **`collect_cmdline_paths(pid, stop_pid, exclude) -> Vec`**: Extract absolute paths from `/proc/{pid}/cmdline` for the process and its ancestor chain. Captures script paths that don't appear in `/proc/{pid}/exe` -- for example, when `#!/usr/bin/env node` runs a script at `/usr/local/bin/claude`, the exe is `/usr/bin/node` but cmdline contains the script path. Paths already in `exclude` (exe-based paths) are omitted. @@ -1272,12 +1328,14 @@ Both IPv4 (`/proc/{pid}/net/tcp`) and IPv6 (`/proc/{pid}/net/tcp6`) tables are c Wraps `tokio::process::Child` + PID. Platform-specific `spawn()` methods delegate to `spawn_impl()`. **Environment setup** (both Linux and non-Linux): + - `OPENSHELL_SANDBOX=1` (always set) - Provider credentials (from `GetSandboxProviderEnvironment` RPC) - Proxy URLs: `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY` (uppercase for curl/wget), `NO_PROXY=127.0.0.1,localhost,::1` for localhost bypass, `http_proxy`, `https_proxy`, `grpc_proxy` (lowercase for gRPC C-core), `no_proxy=127.0.0.1,localhost,::1`, `NODE_USE_ENV_PROXY=1` (required for Node.js built-in `fetch`/`http` clients to honor proxy env vars) - TLS trust store: `NODE_EXTRA_CA_CERTS` (standalone CA cert), `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE` (combined bundle) **Pre-exec closure** (runs in child after fork, before exec -- async-signal-safe): + 1. `setpgid(0, 0)` if non-interactive (create new process group) 2. `setns(fd, CLONE_NEWNET)` to enter network namespace (Linux only) 3. `drop_privileges(policy)`: `initgroups()` -> `setgid()` -> `setuid()` @@ -1288,6 +1346,7 @@ Wraps `tokio::process::Child` + PID. Platform-specific `spawn()` methods delegat ### `drop_privileges()` Resolves user/group names from policy, then: + 1. `initgroups()` to set supplementary groups (Linux only, not macOS) 2. `setgid()` to target group 3. Verify `getegid()` matches the target GID @@ -1313,30 +1372,24 @@ Exit code is `code` if the process exited normally, or `128 + signal` if killed **File:** `crates/openshell-sandbox/src/ssh.rs` -The embedded SSH server provides remote shell access to the sandbox. It uses the `russh` crate and allocates PTYs for interactive sessions. +The embedded SSH server provides remote shell access to the sandbox. It uses the `russh` crate and allocates PTYs for interactive sessions. The daemon listens on a **Unix domain socket** rather than a TCP port -- the gateway never dials the sandbox pod directly. All SSH traffic arrives through the [supervisor session](#supervisor-session)'s `RelayStream` RPC, which the supervisor bridges into the socket. ### Startup -`run_ssh_server()`: -1. Generate an ephemeral Ed25519 host key via `russh::keys::PrivateKey::random()` -2. Bind a `TcpListener` to the configured address -3. Accept connections in a loop, spawning per-connection handlers +`ssh_server_init()` (called from `run_ssh_server()`): -### Handshake verification +1. Generate an ephemeral Ed25519 host key via `russh::keys::PrivateKey::random()`. +2. Ensure the socket's parent directory exists and is owned by root with mode `0700`. The sandbox entrypoint runs as an unprivileged user, so it cannot enter this directory. +3. Remove any stale socket file from a prior run, then `UnixListener::bind(listen_path)`. +4. Set the socket file's mode to `0600` so only the supervisor (root) can connect to it. +5. Signal readiness back to `lib.rs` via a `oneshot` channel. +6. Accept connections in a loop and spawn `handle_connection()` per connection. -Before the SSH protocol begins, the server reads a preface line: +The socket path is taken from `--ssh-socket-path` / `OPENSHELL_SSH_SOCKET_PATH`. The Kubernetes compute driver sets this to `/run/openshell/ssh.sock` by default (see `crates/openshell-driver-kubernetes/src/main.rs`); the VM driver pins it to the same path inside the guest. -``` -NSSH1 {token} {timestamp} {nonce} {hmac_hex}\n -``` +### Access control -`verify_preface()`: -1. Verify magic is `NSSH1` and exactly 5 fields -2. Verify `|now - timestamp|` is within `--ssh-handshake-skew-secs` (default 300s) -3. Compute `HMAC-SHA256(secret, "{token}|{timestamp}|{nonce}")` and compare with `{hmac_hex}` -4. Send `OK\n` on success, `ERR\n` on failure - -This pre-SSH handshake authenticates the gateway-to-sandbox tunnel. After it succeeds, the SSH session uses permissive authentication (`auth_none` and `auth_publickey` both return `Accept`) since the transport is already verified. +The filesystem permissions on the parent directory (`0700`) and the socket itself (`0600`) are the sole authentication boundary. Only the supervisor, which runs as root inside the container, can open the socket. The sandboxed entrypoint process -- dropped to the unprivileged `sandbox` user and further constrained by Landlock -- cannot reach `/run/openshell/` at all. Consequently, the SSH session handler's `auth_none` and `auth_publickey` callbacks both return `Auth::Accept` unconditionally; any byte stream that reaches the daemon has already passed the trust check via the socket's permission bits. ### Shell/exec handling @@ -1351,6 +1404,7 @@ The `SshHandler` implements `russh::server::Handler`: ### PTY child process `spawn_pty_shell()`: + 1. `openpty()` to create a master/slave PTY pair 2. Build `std::process::Command` (not tokio) with slave FDs for stdin/stdout/stderr 3. Set environment: `OPENSHELL_SANDBOX=1`, `HOME=/sandbox`, `USER=sandbox`, `TERM={negotiated}`, proxy URLs, TLS trust store paths, provider credentials @@ -1364,6 +1418,90 @@ The `SshHandler` implements `russh::server::Handler`: - **Reader thread**: Reads from PTY master, sends SSH channel data, sends EOF when done, signals the exit thread - **Exit thread**: Waits for child to exit, waits for reader to finish (ensures correct SSH protocol ordering: data -> EOF -> exit-status -> close), sends exit status and closes the channel +## Supervisor Session + +**File:** `crates/openshell-sandbox/src/supervisor_session.rs` + +The sandbox pod has no inbound network surface. Instead, the supervisor opens a single persistent outbound gRPC stream to the gateway and the gateway uses that stream to request on-demand byte relays back into the sandbox. All SSH connect traffic and `ExecSandbox` calls ride this connection -- there is no reverse HTTP CONNECT, no TCP listener on the pod, and no per-session TLS handshake. + +### Connection model + +```mermaid +sequenceDiagram + participant S as Supervisor (sandbox) + participant GW as Gateway + participant SSHD as Local sshd (Unix socket) + participant Client as Operator / CLI + + S->>GW: ConnectSupervisor stream (mTLS, HTTP/2) + S->>GW: SupervisorMessage::Hello{sandbox_id, instance_id} + GW-->>S: GatewayMessage::SessionAccepted{session_id, heartbeat_interval_secs} + loop Heartbeats (max(accepted.heartbeat_interval_secs, 5)) + S-->>GW: SupervisorHeartbeat + GW-->>S: GatewayHeartbeat + end + + Client->>GW: sandbox connect / ExecSandbox + GW-->>S: GatewayMessage::RelayOpen{channel_id} + S->>GW: RelayStream RPC (new HTTP/2 stream on same Channel) + S->>GW: RelayFrame::Init{channel_id} + S->>SSHD: UnixStream::connect(ssh_socket_path) + loop Relay active + GW-->>S: RelayFrame::Data (raw SSH bytes from operator) + S->>SSHD: write_all + SSHD-->>S: read chunk (up to 16 KiB) + S-->>GW: RelayFrame::Data + end +``` + +One TCP+TLS+HTTP/2 connection carries both the long-lived control stream and every concurrent relay. The sandbox-side `Endpoint` uses `adaptive_window(true)` so HTTP/2 flow control does not throttle bulk transfers (SFTP, `sandbox rsync`) to the 64 KiB default window. + +### Session lifecycle + +`spawn(endpoint, sandbox_id, ssh_socket_path)` launches `run_session_loop()`, which runs for the lifetime of the supervisor: + +1. **Connect**: `grpc_client::connect_channel_pub(endpoint)` builds an mTLS `tonic::transport::Channel`. The same `Channel` is cloned into every subsequent `RelayStream` call so no additional TLS handshakes occur. +2. **Hello**: The supervisor sends `SupervisorMessage::Hello { sandbox_id, instance_id }` as the first envelope, where `instance_id` is a fresh UUID per session. The gateway uses the sandbox ID and instance ID to supersede a stale prior session (see [Supersede](#session-supersede)). +3. **Wait for `SessionAccepted` / `SessionRejected`**: If rejected, the loop returns an error and backs off. On accept, the supervisor clamps `heartbeat_interval_secs` to a minimum of 5 seconds. +4. **Main select loop**: Concurrently reads inbound `GatewayMessage`s and fires heartbeat ticks. Inbound `Heartbeat` messages are acknowledged by the supervisor's outbound heartbeat cadence; `RelayOpen` and `RelayClose` are dispatched to `handle_gateway_message()`. +5. **Reconnect**: Any error in the session (stream error, connect failure, rejected hello) is reported as an OCSF event and the loop sleeps with exponential backoff (`INITIAL_BACKOFF = 1s`, doubled up to `MAX_BACKOFF = 30s`) before redialing. + +### Relay bridge loop + +`handle_gateway_message()` is a synchronous dispatcher. When a `RelayOpen { channel_id }` arrives, it spawns a dedicated task running `handle_relay_open()`. That task: + +1. Creates an outbound `mpsc::channel::(16)` wrapped in a `ReceiverStream`. +2. Sends `RelayFrame { payload: RelayInit { channel_id } }` as the first frame -- this claims the matching pending-relay slot on the gateway. +3. Calls `OpenShellClient::relay_stream(outbound)` on the shared `Channel`. This opens a new HTTP/2 stream on the existing connection -- no new TCP or TLS handshake. +4. `UnixStream::connect(ssh_socket_path)` dials the local sshd. The split read/write halves become the local endpoints of the bridge. +5. Spawns a task that reads from the Unix socket in 16 KiB chunks (`RELAY_CHUNK_SIZE`, matching the default HTTP/2 frame size) and forwards each chunk as `RelayFrame::Data` on the outbound stream. +6. The main loop drains inbound `RelayFrame::Data` messages and writes them to the socket. Non-data inbound frames (e.g. a second `Init`) are treated as protocol errors. +7. On any side closing, the bridge calls `ssh_w.shutdown()` to propagate EOF, drops the outbound sender to close the gRPC stream, and joins the reader task. + +The supervisor has no SSH or HTTP awareness -- it is purely a byte bridge. The protocol on top of the relay is whatever the gateway's caller (interactive `sandbox connect`, `ExecSandbox`, `rsync`-over-ssh) speaks to the sshd. + +### Session supersede + +If the gateway restarts or the sandbox restarts and reconnects with a new `instance_id` for the same `sandbox_id`, the gateway atomically replaces any prior session it has recorded. The new supervisor continues normally; the old stream (if still live on the gateway side) is torn down by the gateway's `remove_if_current` logic. Supervisors never need to coordinate between themselves -- each just keeps trying to connect, and the most recent `Hello` wins. + +If the gateway closes the stream cleanly (`inbound.message()` returns `Ok(None)`), `run_single_session` returns `Ok(())` and a `session_closed` event is emitted. Otherwise the loop reconnects. + +### OCSF telemetry + +Every session and relay transition emits an OCSF `NetworkActivity` event via `ocsf_emit!()` so operators can audit the control-plane connection from the sandbox's own logs. All events are built in `supervisor_session.rs` and covered by unit tests in the `ocsf_event_tests` module. + +| Helper | `activity_id` | `severity` | `status` | Fires when | +|--------|---------------|------------|----------|------------| +| `session_established_event` | `Open` | `Informational` | `Success` | After `SessionAccepted`, includes `session_id` and `heartbeat_secs` in the message | +| `session_closed_event` | `Close` | `Informational` | `Success` | Gateway closed the stream cleanly (`Ok(None)`) | +| `session_failed_event` | `Fail` | `Low` | `Failure` | Connect failed, hello rejected, or stream errored. Includes reconnect attempt counter | +| `relay_open_event` | `Open` | `Informational` | `Success` | `RelayOpen` received from the gateway | +| `relay_closed_event` | `Close` | `Informational` | `Success` | Relay bridge task exited without error | +| `relay_failed_event` | `Fail` | `Low` | `Failure` | Bridge task returned an error (e.g., socket write failure, inbound non-data frame) | +| `relay_close_from_gateway_event` | `Close` | `Informational` | -- | Gateway sent an explicit `RelayClose` on the control stream, with its `reason` | + +The `dst_endpoint` on session events is parsed from the gateway URI by `ocsf_gateway_endpoint()`. Relay events omit a destination (the bridge is sandbox-internal). + ## Zombie Reaping (PID 1 Init Duties) `openshell-sandbox` runs as PID 1 inside the container. In Linux, when a process exits, its parent must call `waitpid()` to collect the exit status; otherwise the process remains as a zombie. Orphaned processes (whose parent exits first) are reparented to PID 1, which becomes responsible for reaping them. @@ -1394,9 +1532,7 @@ This two-phase approach (peek with `WNOWAIT`, then selectively reap) avoids `ECH | `OPENSHELL_LOG_LEVEL` | `--log-level` | `warn` | Log level (trace/debug/info/warn/error) | | `OPENSHELL_POLICY_POLL_INTERVAL_SECS` | | `30` | Poll interval for gRPC policy updates (seconds). Only active in gRPC mode. | | `OPENSHELL_LOG_PUSH_LEVEL` | | `info` | Maximum tracing level for log push to gateway. Events above this level are not streamed. Only active in gRPC mode. | -| `OPENSHELL_SSH_LISTEN_ADDR` | `--ssh-listen-addr` | | SSH server bind address | -| `OPENSHELL_SSH_HANDSHAKE_SECRET` | `--ssh-handshake-secret` | | HMAC secret for SSH handshake | -| `OPENSHELL_SSH_HANDSHAKE_SKEW_SECS` | `--ssh-handshake-skew-secs` | `300` | Allowed clock skew for handshake | +| `OPENSHELL_SSH_SOCKET_PATH` | `--ssh-socket-path` | | Filesystem path to the Unix socket the embedded sshd binds (e.g. `/run/openshell/ssh.sock`). | | `OPENSHELL_INFERENCE_ROUTES` | `--inference-routes` | | Path to YAML inference routes file for standalone routing | ### Injected into child process @@ -1450,8 +1586,8 @@ The sandbox uses `miette` for error reporting and `thiserror` for typed errors. | SSRF: DNS resolution failure | Deny the specific CONNECT request | | Inference route file load/parse error | Fatal -- sandbox startup aborts | | Inference route file with empty routes | Inference routing disabled (graceful) | -| Inference cluster bundle with empty routes | Inference routing stays enabled with empty cache; refresh can activate routes later | -| Inference cluster bundle fetch failure | Warn + inference routing disabled (graceful) | +| Inference gateway bundle with empty routes | Inference routing stays enabled with empty cache; refresh can activate routes later | +| Inference gateway bundle fetch failure | Warn + inference routing disabled (graceful) | | Inference interception: missing InferenceContext | Denied outcome + structured CONNECT deny log | | Inference interception: missing TLS state | Denied outcome + structured CONNECT deny log | | Inference interception: TLS handshake failure | Denied outcome + structured CONNECT deny log | @@ -1472,20 +1608,31 @@ The sandbox uses `miette` for error reporting and `thiserror` for typed errors. | Credential injection: resolved value contains CR/LF/null | Placeholder treated as unresolvable, fail-closed | | Credential injection: path credential contains traversal/separator | HTTP 500, connection closed (fail-closed) | | Credential injection: percent-encoded placeholder bypass attempt | HTTP 500, connection closed (fail-closed) | -| L7 parse error | Close the connection | -| SSH server failure | Async task error logged, main process unaffected | +| L7 parse or path-canonicalization error | Emit denied OCSF network event with `status_detail`, close the connection | +| SSH socket bind failure | Fatal -- reported through the readiness channel and aborts startup | +| SSH server accept failure | Async task error logged, main process unaffected | +| Supervisor session: connect failure | Emit `session_failed` OCSF event, sleep with exponential backoff (1s -> 30s) and reconnect | +| Supervisor session: `SessionRejected` | Emit `session_failed` event with rejection reason; backoff and reconnect | +| Supervisor session: stream error mid-session | Emit `session_failed` event; backoff and reconnect | +| Supervisor session: gateway closes stream cleanly | Emit `session_closed` event and exit the task (no reconnect) | +| Relay bridge: `RelayStream` RPC failure | Emit `relay_failed` event; the individual relay is abandoned, the session stays up | +| Relay bridge: Unix socket connect failure | Emit `relay_failed` event; gateway observes EOF on the RelayStream | +| Relay bridge: non-data inbound frame after Init | Emit `relay_failed` event with protocol error | | Process timeout | Kill process, return exit code 124 | ## Logging Dual-output logging is configured in `main.rs`: + - **stdout**: Filtered by `--log-level` (default `warn`), uses ANSI colors - **`/var/log/openshell.log`**: Fixed at `info` level, no ANSI, non-blocking writer Key structured log events: + - `CONNECT`: One per proxy CONNECT request (for non-`inference.local` targets) with full identity context. Inference interception failures produce a separate `info!()` log with `action=deny` and the denial reason. - `BYPASS_DETECT`: One per detected direct connection attempt that bypassed the HTTP CONNECT proxy. Includes destination, protocol, process identity (best-effort), and remediation hint. Emitted at `warn` level. - `L7_REQUEST`: One per L7-inspected request with method, path, and decision +- Supervisor session / relay OCSF events: `session_established`, `session_closed`, `session_failed`, `relay_open`, `relay_closed`, `relay_failed`, `relay_close_from_gateway` (see [Supervisor Session](#supervisor-session)). - Sandbox lifecycle events: process start, exit, namespace creation/cleanup, bypass rule installation - Policy reload events: new version detected, reload success/failure, status report outcomes @@ -1516,6 +1663,7 @@ flowchart LR ``` Two log sources feed the same `TracingLogBus`: + - **Gateway logs** (`source: "gateway"`): Generated by the server's `SandboxLogLayer` tracing layer when server-side code emits events containing a `sandbox_id` field. These capture reconciliation, provisioning, and management operations. - **Sandbox logs** (`source: "sandbox"`): Pushed from the sandbox supervisor via the `PushSandboxLogs` client-streaming RPC. These capture proxy decisions, policy reloads, process lifecycle, and all other sandbox-internal tracing events. @@ -1534,6 +1682,7 @@ pub struct LogPushLayer { ``` Key behaviors: + - **Level filtering**: Defaults to `INFO`. Configurable via the `OPENSHELL_LOG_PUSH_LEVEL` environment variable (accepts `trace`, `debug`, `info`, `warn`, `error`). Events above the configured level are silently discarded. - **Best-effort delivery**: Uses `try_send()` on the mpsc channel. If the channel is full (1024 lines buffered), the event is dropped. Logging never blocks the sandbox supervisor. - **Structured fields**: Implements a `LogVisitor` that collects all tracing key-value fields (e.g., `dst_host`, `action`, `policy`) into a `HashMap`. The `message` field is extracted separately; all other fields go into `SandboxLogLine.fields`. @@ -1570,6 +1719,7 @@ The background task batches log lines and streams them to the gateway: **File:** `crates/openshell-server/src/grpc.rs` (`push_sandbox_logs`) The `PushSandboxLogs` RPC handler processes each batch: + 1. Validates `sandbox_id` is non-empty (skips empty batches). 2. Iterates over `batch.logs`, capped at 100 lines per batch to prevent abuse. 3. Forces `log.source = "sandbox"` on every line -- the sandbox cannot claim to be the gateway. @@ -1581,6 +1731,7 @@ The `PushSandboxLogs` RPC handler processes each batch: **File:** `crates/openshell-server/src/tracing_bus.rs` `publish_external()` wraps the `SandboxLogLine` in a `SandboxStreamEvent` and calls the internal `publish()` method, which: + 1. Sends the event to the per-sandbox `broadcast::Sender` (capacity 1024). Subscribers (active `WatchSandbox` streams) receive the event immediately. 2. Appends the event to the per-sandbox tail buffer (`VecDeque`), capped at 2000 lines. Overflow evicts the oldest entry. @@ -1654,12 +1805,13 @@ Filtering is implemented server-side. For `WatchSandbox`, filters apply to both `print_log_line()` in `crates/openshell-cli/src/run.rs` formats each log line: -``` +```text [timestamp] [source ] [level] [target] message key=value key=value ``` Example output: -``` + +```text [1708891234.567] [sandbox] [INFO ] [openshell_sandbox::proxy] CONNECT api.example.com:443 dst_host=api.example.com action=allow [1708891234.890] [gateway] [INFO ] [openshell_server::grpc] ReportPolicyStatus: sandbox reported policy load result ``` diff --git a/architecture/security-policy.md b/architecture/security-policy.md index b8bda8f91a..1445705db5 100644 --- a/architecture/security-policy.md +++ b/architecture/security-policy.md @@ -31,7 +31,7 @@ The YAML data file is preprocessed before loading into the OPA engine: L7 polici ### gRPC Mode (Production) -When the sandbox runs inside a managed cluster, it fetches its typed protobuf policy from the gateway: +When the sandbox runs under a gateway-managed compute platform, it fetches its typed protobuf policy from the gateway: ```bash openshell-sandbox \ @@ -162,6 +162,24 @@ This guarantees that the same logical policy always produces the same hash regar **Idempotent updates**: `UpdateSandboxPolicy` compares the deterministic hash of the submitted policy against the latest stored revision's hash. If they match, the handler returns the existing version and hash without creating a new revision. The CLI detects this (the returned version equals the pre-call version) and prints `Policy unchanged` instead of `Policy version N submitted`. This makes repeated `policy set` calls safe and idempotent. +### Incremental Merge Updates + +`UpdateConfigRequest.merge_operations` supports batched incremental changes to the dynamic `network_policies` section. The CLI exposes this as `openshell policy update`. + +Supported first-pass operations: + +- `--add-endpoint host:port[:access[:protocol[:enforcement]]]` +- `--remove-endpoint host:port` +- `--remove-rule ` +- `--add-allow host:port:METHOD:path_glob` +- `--add-deny host:port:METHOD:path_glob` + +`--add-allow` and `--add-deny` target existing `protocol: rest` endpoints only. `--binary` may be repeated with `--add-endpoint`, and `--rule-name` is allowed only when exactly one `--add-endpoint` is present. + +Each `openshell policy update` invocation is atomic at the revision level: the CLI sends one `merge_operations` batch, the server merges the whole batch into the latest policy, validates the result, and persists at most one new revision. Concurrency is handled with optimistic retries on the `(sandbox_id, version)` uniqueness boundary. If another writer wins first, the server refetches the latest policy, reapplies the full batch, revalidates it, and retries. This preserves batch atomicity without serializing all sandbox policy writes behind a sandbox-global mutex. + +The gateway emits per-sandbox OCSF `CONFIG:*` audit lines when incremental merge operations are applied and when draft chunks are approved or removed. These audit lines are streamed through the existing gateway log path, so operators can inspect the exact logical mutation that produced a policy revision without waiting for the sandbox poll loop to reload that revision. + ### Policy Revision Statuses | Status | Meaning | @@ -185,7 +203,7 @@ The poll loop: 2. Fetches the current policy via `GetSandboxSettings`, which returns the latest version, its policy payload, and a SHA-256 hash. 3. Compares the returned version against the locally tracked `current_version`. If the server version is not greater, the loop sleeps and retries. 4. On a new version, calls `OpaEngine::reload_from_proto()` which builds a complete new `regorus::Engine` through the same validated pipeline as the initial load (proto-to-JSON conversion, L7 validation, access preset expansion). -5. If the new engine builds successfully, it atomically replaces the inner `Mutex`. If it fails, the previous engine is untouched. +5. If the new engine builds successfully, it atomically replaces the inner `Mutex` and increments the policy generation. Active L7 keep-alive tunnels close before forwarding another request after they observe the new generation. If reload fails, the previous engine and generation are untouched. 6. Reports success or failure back to the server via `ReportPolicyStatus`. See `crates/openshell-sandbox/src/grpc_client.rs` -- `CachedOpenShellClient`. @@ -206,9 +224,20 @@ Failure scenarios that trigger LKG behavior include: ### CLI Commands -The `openshell policy` subcommand group manages live policy updates: +The `openshell policy` subcommand group manages live policy updates through full replacement (`policy set`) and incremental merges (`policy update`): ```bash +# Merge endpoint/rule changes into the current sandbox policy +openshell policy update \ + --add-endpoint api.github.com:443:read-only:rest:enforce \ + --binary /usr/bin/gh \ + --wait + +# Add a REST allow rule to an existing endpoint +openshell policy update \ + --add-allow api.github.com:443:POST:/repos/*/issues \ + --wait + # Push a new policy to a running sandbox openshell policy set --policy updated-policy.yaml @@ -254,7 +283,9 @@ The `--global` flag on `policy set`, `policy delete`, `policy list`, and `policy Both `set` and `delete` require interactive confirmation (or `--yes` to bypass). The `--wait` flag is rejected for global policy updates: `"--wait is not supported for global policies; global policies are effective immediately"`. When a global policy is active, sandbox-scoped policy mutations are blocked: + - `policy set ` returns `FailedPrecondition: "policy is managed globally"` +- `policy update ` returns `FailedPrecondition: "policy is managed globally"` - `rule approve`, `rule approve-all` return `FailedPrecondition: "cannot approve rules while a global policy is active"` - Revoking a previously approved draft chunk is blocked (it would modify the sandbox policy) - Rejecting pending chunks is allowed (does not modify the sandbox policy) @@ -270,7 +301,7 @@ See [Gateway Settings Channel](gateway-settings.md#global-policy-lifecycle) for When `--full` is specified, the server includes the deserialized `SandboxPolicy` protobuf in the `SandboxPolicyRevision.policy` field (see `crates/openshell-server/src/grpc.rs` -- `policy_record_to_revision()` with `include_policy: true`). The CLI converts this proto back to YAML via `policy_to_yaml()`, which uses a `BTreeMap` for `network_policies` to produce deterministic key ordering. See `crates/openshell-cli/src/run.rs` -- `policy_to_yaml()`, `policy_get()`. -See `crates/openshell-cli/src/main.rs` -- `PolicyCommands` enum, `crates/openshell-cli/src/run.rs` -- `policy_set()`, `policy_get()`, `policy_list()`. +See `crates/openshell-cli/src/main.rs` -- `PolicyCommands` enum, `crates/openshell-cli/src/run.rs` -- `policy_update()`, `policy_set()`, `policy_get()`, `policy_list()`. --- @@ -363,7 +394,7 @@ Controls Landlock LSM compatibility behavior. **Static field** -- immutable afte **Per-path error handling**: `PathFd::new()` (which wraps `open(path, O_PATH | O_CLOEXEC)`) can fail for several reasons beyond path non-existence: `EACCES` (permission denied), `ELOOP` (symlink loop), `ENAMETOOLONG`, `ENOTDIR`. Each failure is classified with a human-readable reason in logs. In `best_effort` mode, the path is skipped and ruleset construction continues. In `hard_requirement` mode, the error is fatal. -**Baseline path filtering**: The enrichment functions (`enrich_proto_baseline_paths`, `enrich_sandbox_baseline_paths`) pre-filter system-injected baseline paths (e.g., `/app`) by checking `Path::exists()` before adding them to the policy. This prevents missing baseline paths from reaching Landlock at all. User-specified paths are not pre-filtered — they are evaluated at Landlock apply time so that misconfigurations surface as warnings (`best_effort`) or errors (`hard_requirement`). +**Baseline path filtering**: The enrichment functions (`enrich_proto_baseline_paths`, `enrich_sandbox_baseline_paths`) pre-filter system-injected baseline paths (e.g., `/app`) by checking `Path::exists()` before adding them to the policy. This prevents missing baseline paths from reaching Landlock at all. If a baseline `read_write` path is explicitly configured in `read_only`, enrichment skips the promotion and preserves the stricter policy intent. User-specified paths are not pre-filtered — they are evaluated at Landlock apply time so that misconfigurations surface as warnings (`best_effort`) or errors (`hard_requirement`). **Zero-rule safety check**: If all paths in the ruleset fail to open, `apply()` returns an error rather than calling `restrict_self()` on an empty ruleset. An empty Landlock ruleset with `restrict_self()` would block all filesystem access — the inverse of the intended degradation behavior. This error is caught by the outer `BestEffort` handler, which logs a warning and continues without Landlock. @@ -438,12 +469,17 @@ Each endpoint defines a network destination and, optionally, L7 inspection behav | `host` | `string` | _(required)_ | Hostname or glob pattern to match (case-insensitive). Supports wildcards (`*.example.com`). Optional when `allowed_ips` is set (see [Hostless Endpoints](#hostless-endpoints-allowed_ips-without-host)). See [Host Wildcards](#host-wildcards). | | `port` | `integer` | _(required)_ | TCP port to match. Mutually exclusive with `ports` — if both are set, `ports` takes precedence. See [Multi-Port Endpoints](#multi-port-endpoints). | | `ports` | `integer[]`| `[]` | Multiple TCP ports to match. When non-empty, the endpoint covers all listed ports. Backwards compatible with `port`. See [Multi-Port Endpoints](#multi-port-endpoints). | +| `path` | `string` | `""` | Optional HTTP path glob for L7 endpoint selection when multiple protocols share a host:port, such as `/repos/**` and `/graphql`. Empty matches all paths. | | `protocol` | `string` | `""` | Application protocol for L7 inspection. See [Behavioral Trigger: L7 Inspection](#behavioral-trigger-l7-inspection). | | `tls` | `string` | `""` (auto) | TLS handling mode. Absent or empty: auto-detect and terminate TLS if detected. `"skip"`: bypass TLS detection entirely. `"terminate"` and `"passthrough"` are deprecated (treated as auto). See [Behavioral Trigger: TLS Handling](#behavioral-trigger-tls-handling). | | `enforcement` | `string` | `"audit"` | L7 enforcement mode: `"enforce"` or `"audit"` | | `access` | `string` | `""` | Shorthand preset for common L7 rule sets. Mutually exclusive with `rules`. | | `rules` | `L7Rule[]` | `[]` | Explicit L7 allow rules. Mutually exclusive with `access`. | | `allowed_ips` | `string[]` | `[]` | IP allowlist for SSRF override. Entries overlapping always-blocked ranges (loopback, link-local, unspecified) are rejected at load time. See [Private IP Access via `allowed_ips`](#private-ip-access-via-allowed_ips). | +| `allow_encoded_slash` | `bool` | `false` | Preserves `%2F` inside L7 request path segments instead of rejecting the request. Required for endpoints such as npm scoped packages. | +| `persisted_queries` | `string` | `"deny"` | GraphQL hash-only/saved-query behavior. Use `"allow_registered"` only with `graphql_persisted_queries`. | +| `graphql_persisted_queries` | `map` | `{}` | Trusted GraphQL persisted-query registry keyed by hash or service-specific ID. Values contain `operation_type`, optional `operation_name`, and optional root `fields`. | +| `graphql_max_body_bytes` | `integer` | `65536` | Maximum GraphQL request body size buffered for inspection. Larger GraphQL bodies are rejected before policy evaluation. | #### `NetworkBinary` @@ -475,6 +511,13 @@ rules: query: labels: any: ["bug*", "p1*"] + - allow: + operation_type: query + fields: [viewer, repository] + - allow: + operation_type: mutation + operation_name: Issue* + fields: [createIssue] ``` #### `L7Allow` @@ -485,18 +528,23 @@ rules: | `path` | `string` | URL path glob pattern: `**` matches everything, otherwise `glob.match` with `/` delimiter. | | `command` | `string` | SQL command: `SELECT`, `INSERT`, `UPDATE`, `DELETE`, or `*` (any). Case-insensitive matching. For `protocol: sql` endpoints. | | `query` | `map` | Optional REST query rules keyed by decoded query param name. Value is either a glob string (for example, `tag: "foo-*"`) or `{ any: ["foo-*", "bar-*"] }`. | +| `operation_type` | `string` | GraphQL operation type: `query`, `mutation`, `subscription`, or `*`. Required for `protocol: graphql` allow rules. | +| `operation_name` | `string` | Optional GraphQL operation-name glob. Omit to match any operation name. | +| `fields` | `string[]` | Optional GraphQL root-field globs. For allow rules, every selected root field must match one configured glob. For deny rules, any matching root field blocks the request. | -Method and command fields use `*` as wildcard for "any". Path patterns use `**` for "match everything" and standard glob patterns with `/` as a delimiter otherwise. Query matching is case-sensitive and evaluates decoded values; when duplicate keys are present in the request, every value for that key must match the configured matcher. See `sandbox-policy.rego` -- `method_matches()`, `path_matches()`, `command_matches()`, `query_params_match()`. +Method, command, and GraphQL operation type fields use `*` as wildcard for "any". Path patterns use `**` for "match everything" and standard glob patterns with `/` as a delimiter otherwise. Query matching is case-sensitive and evaluates decoded values; when duplicate keys are present in the request, every value for that key must match the configured matcher. GraphQL field and operation-name matching also uses glob patterns. See `sandbox-policy.rego` -- `method_matches()`, `path_matches()`, `command_matches()`, `query_params_match()`, and `graphql_*`. + +GraphQL inspection supports `GET` and `POST` GraphQL-over-HTTP envelopes, JSON batches, named-operation selection, fragments at the operation root, Apollo persisted-query hashes, and service-specific saved-query IDs (`id`, `documentId`, or `queryId`). Hash-only or saved-query-only requests have no parseable document, so they are denied unless `persisted_queries: allow_registered` is set and the hash or ID appears in `graphql_persisted_queries`. If a batch contains any denied, malformed, or unregistered operation, the whole request is denied. #### Access Presets The `access` field provides shorthand for common rule sets. During preprocessing, presets are expanded into explicit `rules` arrays before Rego evaluation. -| Preset | Expands To | Description | -| ------------ | ------------------------------------------------------------------ | ---------------------------------------- | -| `read-only` | `GET/**`, `HEAD/**`, `OPTIONS/**` | Safe read-only HTTP methods on all paths | -| `read-write` | `GET/**`, `HEAD/**`, `OPTIONS/**`, `POST/**`, `PUT/**`, `PATCH/**` | Read and write but not delete | -| `full` | `*/**` | All methods, all paths | +| Preset | REST expansion | GraphQL expansion | Description | +| ------------ | ------------------------------------------------------------------ | ------------------------------ | ---------------------------------------- | +| `read-only` | `GET/**`, `HEAD/**`, `OPTIONS/**` | `operation_type: query` | Safe read-only access | +| `read-write` | `GET/**`, `HEAD/**`, `OPTIONS/**`, `POST/**`, `PUT/**`, `PATCH/**` | `query`, `mutation` | Read and write but not delete for REST | +| `full` | `*/**` | `operation_type: "*"` | All supported actions | See `crates/openshell-sandbox/src/l7/mod.rs` -- `expand_access_presets()`. @@ -633,7 +681,7 @@ network_policies: Inference routing to `inference.local` is handled by the proxy's `InferenceContext`, not by the OPA policy engine or an `inference` block in the policy YAML. The proxy intercepts HTTPS CONNECT requests to `inference.local` and routes matching inference API requests (e.g., `POST /v1/chat/completions`, `POST /v1/messages`) through the sandbox-local `openshell-router`. See [Inference Routing](inference-routing.md) for details on route configuration and the router architecture. -The proxy always runs in proxy mode so that `inference.local` is addressable from within the sandbox's network namespace. Inference route sources are configured separately from policy: via `--inference-routes` (file mode) or fetched from the gateway's inference bundle (cluster mode). See `crates/openshell-sandbox/src/proxy.rs` -- `InferenceContext`, `crates/openshell-sandbox/src/l7/inference.rs`. +The proxy always runs in proxy mode so that `inference.local` is addressable from within the sandbox's network namespace. Inference route sources are configured separately from policy: via `--inference-routes` (file mode) or fetched from the gateway's inference bundle (gateway mode). See `crates/openshell-sandbox/src/proxy.rs` -- `InferenceContext`, `crates/openshell-sandbox/src/l7/inference.rs`. --- @@ -687,11 +735,16 @@ flowchart LR | -------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `protocol` absent or empty | **L4 (transport)** | The proxy performs a raw `copy_bidirectional` after the CONNECT handshake. No application-layer inspection occurs. Only the host:port and binary identity are checked. | | `protocol: rest` | **L7 (application)** | The proxy parses each HTTP/1.1 request within the tunnel, evaluates method+path against the endpoint's `rules`, and either forwards or denies each request individually. | +| `protocol: graphql` | **L7 (application)** | The proxy parses GraphQL-over-HTTP requests, classifies operation type, operation name, root fields, and persisted-query identifiers, then evaluates GraphQL allow and deny rules. | | `protocol: sql` | **L7 (application, audit-only)** | Reserved for SQL protocol inspection. Currently falls through to passthrough with a warning. `enforcement: enforce` is rejected at validation time for SQL endpoints. | -This is the single most important behavioral trigger in the policy language. An endpoint with no `protocol` field passes traffic opaquely after the L4 (CONNECT) check. Adding `protocol: rest` activates per-request HTTP parsing and policy evaluation inside the proxy. +This is the single most important behavioral trigger in the policy language. An endpoint with no `protocol` field passes traffic opaquely after the L4 (CONNECT) check. Adding `protocol: rest` or `protocol: graphql` activates per-request HTTP parsing and policy evaluation inside the proxy. + +**Implementation path**: After L4 CONNECT is allowed, the proxy calls `query_l7_route_snapshot()` which evaluates the Rego rule `data.openshell.sandbox._matching_endpoint_configs` and records the policy generation. If one or more endpoint `protocol` configs are returned, the proxy enters path-aware L7 route selection instead of `copy_bidirectional()`. See `crates/openshell-sandbox/src/proxy.rs` -- `handle_tcp_connection()`. + +For L7-inspected CONNECT tunnels, the proxy binds endpoint config and the per-tunnel policy engine clone to the policy generation observed at tunnel setup. If a live policy reload advances the generation, the relay closes the existing keep-alive tunnel before forwarding another request. HTTP passthrough tunnels without endpoint `protocol` use the same generation guard for parsed requests even though they do not evaluate L7 OPA rules. Clients should reconnect so the next request is evaluated under the current policy. -**Implementation path**: After L4 CONNECT is allowed, the proxy calls `query_l7_config()` which evaluates the Rego rule `data.openshell.sandbox.matched_endpoint_config`. This rule only matches endpoints that have a `protocol` field set (see `sandbox-policy.rego` line `ep.protocol`). If a config is returned, the proxy enters `relay_with_inspection()` instead of `copy_bidirectional()`. See `crates/openshell-sandbox/src/proxy.rs` -- `handle_tcp_connection()`. +Raw streams are connection-scoped and outside L7 live-reload guarantees. This includes endpoints with `tls: skip`, non-HTTP CONNECT payloads, SQL audit fallback passthrough, HTTP upgrades after `101 Switching Protocols`, and already-forwarded streaming response bodies such as SSE. A policy reload applies to the next connection or next parsed HTTP request; it does not terminate raw bytes already relayed outside the HTTP request parser. **Validation requirement**: When `protocol` is set, either `rules` or `access` must also be present. An endpoint with `protocol` but no rules/access is rejected at validation time because it would deny all traffic (no allow rules means nothing matches). See `crates/openshell-sandbox/src/l7/mod.rs` -- `validate_l7_policies()`. @@ -726,15 +779,15 @@ If any condition fails, the proxy returns `403 Forbidden`. 5. Resolves DNS and validates all IPs are private and within `allowed_ips` 6. Connects to upstream 7. Rewrites the request: absolute-form → origin-form (`GET /path HTTP/1.1`), strips hop-by-hop headers, adds `Via: 1.1 openshell-sandbox` and `Connection: close` -8. Forwards the rewritten request, then relays bidirectionally using `tokio::io::copy_bidirectional` (supports chunked transfer, SSE streams, and other long-lived responses with no idle timeout) +8. Relays the rewritten request and response through the shared guarded HTTP relay. This reuses the same request body framing, CL/TE rejection, credential rewrite fail-closed behavior, unsolicited `101` blocking, and policy-generation checks as CONNECT L7 HTTP. -**V1 simplifications**: Forward proxy v1 injects `Connection: close` (no keep-alive). Every forward proxy connection handles exactly one request-response exchange. When an endpoint has L7 rules configured, the forward proxy evaluates the single request's method and path against L7 policy before forwarding. +**V1 simplifications**: Forward proxy v1 injects `Connection: close` (no keep-alive). Every forward proxy connection handles exactly one request-response exchange. The request is bound to the policy generation used for the L4 allow decision and is checked again before upstream connect and request forwarding. When an endpoint has L7 rules configured, the forward proxy also evaluates the single request's method and path against L7 policy before forwarding. **Implementation**: See `crates/openshell-sandbox/src/proxy.rs` -- `handle_forward_proxy()`, `parse_proxy_uri()`, `rewrite_forward_request()`. **Logging**: Forward proxy requests are logged distinctly from CONNECT: -``` +```text FORWARD method=GET dst_host=10.86.8.223 dst_port=8000 path=/screenshot/ action=allow policy=computer-control ``` @@ -753,7 +806,7 @@ flowchart TD K -- No --> L["403 Forbidden"] K -- Yes --> M["TCP connect to upstream"] M --> N["Rewrite request to origin-form
Add Via + Connection: close"] - N --> O["Forward request + copy_bidirectional"] + N --> O["Guarded HTTP relay"] ``` #### Example: Forward Proxy Policy @@ -793,8 +846,8 @@ TLS termination is automatic. The proxy peeks the first bytes of every CONNECT t | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tls` absent or `""` (default) | **Auto-detect**: The proxy peeks the first bytes of the tunnel. If TLS is detected (ClientHello pattern), the proxy terminates TLS transparently (MITM), enabling credential injection and L7 inspection. If plaintext HTTP is detected, the proxy inspects directly. If neither, traffic is relayed raw. | | `tls: "skip"` | **Explicit opt-out**: No TLS detection, no termination, no credential injection. The tunnel is a raw `copy_bidirectional` relay. Use for client-cert mTLS to upstream or non-standard binary protocols. | -| `tls: "terminate"` *(deprecated)* | Treated as auto-detect. Emits a deprecation warning: "TLS termination is now automatic. Use `tls: skip` to explicitly disable." | -| `tls: "passthrough"` *(deprecated)* | Treated as auto-detect. Emits the same deprecation warning. | +| `tls: "terminate"` _(deprecated)_ | Treated as auto-detect. Emits a deprecation warning: "TLS termination is now automatic. Use `tls: skip` to explicitly disable." | +| `tls: "passthrough"` _(deprecated)_ | Treated as auto-detect. Emits the same deprecation warning. | **Prerequisites for TLS termination (auto-detect path)**: @@ -804,9 +857,10 @@ TLS termination is automatic. The proxy peeks the first bytes of every CONNECT t **Certificate caching**: Per-hostname leaf certificates are cached (up to 256 entries, then the entire cache is cleared). See `crates/openshell-sandbox/src/l7/tls.rs` -- `CertCache`. -**Credential injection**: When TLS is auto-terminated but no L7 policy is configured (no `protocol` field), the proxy enters a passthrough relay that rewrites credential placeholders in HTTP headers (via `SecretResolver`) and logs requests for observability, but does not evaluate L7 OPA rules. This means credential injection works on all HTTPS endpoints automatically. +**Credential injection**: When TLS is auto-terminated but no L7 policy is configured (no `protocol` field), the proxy enters a passthrough relay that rewrites credential placeholders in HTTP headers (via `SecretResolver`) and logs requests for observability, but does not evaluate L7 OPA rules. The relay still closes parsed keep-alive HTTP tunnels after policy generation changes before forwarding another request. This means credential injection works on all HTTPS endpoints automatically. **Validation warnings**: + - `tls: terminate` or `tls: passthrough`: deprecated, emits a warning. - `tls: skip` with `protocol: rest` on port 443: emits a warning ("L7 inspection cannot work on encrypted traffic"). @@ -939,9 +993,11 @@ sequenceDiagram OPA-->>Proxy: allowed=true, matched_policy="api_policy" Proxy-->>Client: 200 Connection Established - Note over Proxy: Query L7 config for matched endpoint - Proxy->>OPA: query_endpoint_config(host, port, binary) - OPA-->>Proxy: {protocol: rest, enforcement: enforce} + Note over Proxy: Query L7 config and generation for matched endpoint + Proxy->>OPA: query_endpoint_config_with_generation(host, port, binary) + OPA-->>Proxy: {protocol: rest, enforcement: enforce}, generation=N + Proxy->>OPA: clone_engine_for_tunnel(generation=N) + OPA-->>Proxy: generation-bound tunnel evaluator Note over Proxy: Auto-detect TLS (peek first bytes) Note over Proxy: TLS ClientHello detected → terminate @@ -955,9 +1011,12 @@ sequenceDiagram loop Per HTTP request in tunnel Client->>Proxy: GET /repos/myorg/foo HTTP/1.1 + Note over Proxy: Close if policy generation changed Note over Proxy: Parse HTTP request line + headers + Note over Proxy: Close if policy generation changed Proxy->>OPA: allow_request(method=GET, path=/repos/myorg/foo) OPA-->>Proxy: allowed=true + Note over Proxy: Close if policy generation changed Proxy->>Upstream: GET /repos/myorg/foo HTTP/1.1 Upstream-->>Proxy: 200 OK (response body) Proxy-->>Client: 200 OK (response body) @@ -1074,9 +1133,9 @@ IP classification helpers live in `crates/openshell-core/src/net.rs` and are sha Runtime resolution and enforcement functions remain in `crates/openshell-sandbox/src/proxy.rs`: -- **`resolve_and_reject_internal(host, port) -> Result, String>`**: Default SSRF check. Resolves DNS via `tokio::net::lookup_host()`, then checks every resolved address against `is_internal_ip()`. If any address is internal, the entire connection is rejected. +- **`resolve_and_reject_internal(host, port, entrypoint_pid) -> Result, String>`**: Default SSRF check. Resolves the host using the sandbox's `/etc/hosts` first on Linux (via `/proc//root/etc/hosts`, which captures Kubernetes `hostAliases`), then falls back to `tokio::net::lookup_host()`. It checks every resolved address against `is_internal_ip()`. If any address is internal, the entire connection is rejected. -- **`resolve_and_check_allowed_ips(host, port, allowed_ips) -> Result, String>`**: Allowlist-based SSRF check. Resolves DNS, rejects any always-blocked IPs, then verifies every resolved address matches at least one entry in the `allowed_ips` list. +- **`resolve_and_check_allowed_ips(host, port, allowed_ips, entrypoint_pid) -> Result, String>`**: Allowlist-based SSRF check. Resolves the host using the sandbox's `/etc/hosts` first on Linux, rejects any always-blocked IPs, then verifies every resolved address matches at least one entry in the `allowed_ips` list. - **`parse_allowed_ips(raw) -> Result, String>`**: Parses CIDR/IP strings into typed `IpNet` values. **Rejects entries at load time** that overlap always-blocked ranges (loopback, link-local, unspecified) via `is_always_blocked_net`. Accepts both CIDR notation (`10.0.5.0/24`) and bare IPs (`10.0.5.20`, treated as `/32`). This prevents confusing UX where an entry is accepted in policy but silently denied at runtime. @@ -1132,6 +1191,8 @@ The `allowed_ips` field on a `NetworkEndpoint` enables controlled access to priv **Load-time validation**: `parse_allowed_ips` rejects entries that overlap always-blocked ranges with a hard error at policy load time. This catches misconfigurations early — an entry like `127.0.0.0/8` or `0.0.0.0/0` in `allowed_ips` would be silently un-enforceable at runtime, so it is rejected before the policy is applied. The same validation runs in both file mode (sandbox startup) and gRPC mode (live policy updates via `OpaEngine::reload_from_proto`). +**Sandbox `/etc/hosts` and `hostAliases`**: On Linux, the proxy consults the sandbox's `/etc/hosts` before falling back to DNS. This gives policy evaluation the same hostname-to-IP view that sandboxed tools see when Kubernetes `hostAliases` populate `/etc/hosts`. This is resolver input only. It does **not** synthesize `allowed_ips`, and it does not bypass the private-IP SSRF check. If `searxng.local` resolves to `192.168.1.105`, the request still needs `allowed_ips: ["192.168.1.105/32"]` to succeed. + **Implicit `allowed_ips` for IP hosts**: When a policy endpoint has a literal IP address as its host (e.g., `host: 10.0.5.20`), the proxy synthesizes an `allowed_ips` entry automatically via `implicit_allowed_ips_for_ip_host`. If the host is an always-blocked address (e.g., `127.0.0.1`, `169.254.169.254`, `0.0.0.0`), the function returns empty and logs a warning — no `allowed_ips` entry is synthesized, so the standard SSRF rejection applies. This supports three usage modes: @@ -1142,9 +1203,27 @@ This supports three usage modes: | **Host + allowlist** | `host` + `allowed_ips` | Domain must match `host` AND resolve to an IP in `allowed_ips` | | **Hostless allowlist** | `allowed_ips` only (no `host`) | Any domain allowed on the specified `port`, as long as it resolves to an IP in `allowed_ips` | +Example: + +```yaml +network_policies: + websearch: + name: websearch + endpoints: + - host: searxng.local + port: 8080 + allowed_ips: + - "192.168.1.105/32" + binaries: + - path: /usr/bin/curl +``` + +With a matching sandbox `/etc/hosts` entry such as `192.168.1.105 searxng.local`, this policy works. Without the `allowed_ips` entry, the request stays blocked because the resolved destination is private. + #### `allowed_ips` Format Entries can be: + - **CIDR notation**: `10.0.5.0/24`, `172.16.0.0/12`, `192.168.1.0/24` - **Exact IP**: `10.0.5.20` (treated as `/32` for IPv4 or `/128` for IPv6) @@ -1410,14 +1489,14 @@ Evaluated on every CONNECT request and every forward proxy request. The same OPA | `network_action` | Same input | `"allow"` if endpoint + binary matched, `"deny"` otherwise | | `deny_reason` | Same input | Human-readable string explaining why access was denied | | `matched_network_policy` | Same input | Name of the matched policy (for audit logging) | -| `matched_endpoint_config` | Same input | Raw endpoint object for L7 config extraction (returned if endpoint has `protocol` or `allowed_ips` field) | +| `matched_endpoint_config` | Same input | Raw endpoint object for L7 config extraction (returned if endpoint has `protocol`, `allowed_ips`, or explicit TLS config) | ### L7 Rules (per-request within tunnel) -| Rule | Signature | Returns | -| --------------------- | ------------------------------------------------------------------------------- | -------------------------------------------------------------- | -| `allow_request` | `input.network.*`, `input.exec.*`, `input.request.method`, `input.request.path` | `true` if the request matches any rule in the matched endpoint | -| `request_deny_reason` | Same input | Human-readable deny message | +| Rule | Signature | Returns | +| --------------------- | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| `allow_request` | `input.network.*`, `input.exec.*`, `input.request.method`, `input.request.path`, optional `request.graphql` | `true` if the request matches the matched endpoint's L7 rules | +| `request_deny_reason` | Same input | Human-readable deny message | See `sandbox-policy.rego` for the full Rego implementation. diff --git a/architecture/system-architecture.md b/architecture/system-architecture.md index 5ea92064ed..b271cdd677 100644 --- a/architecture/system-architecture.md +++ b/architecture/system-architecture.md @@ -9,35 +9,35 @@ graph TB CLI["OpenShell CLI
(openshell)"] TUI["OpenShell TUI
(openshell term)"] SDK["Python SDK
(openshell)"] - LocalConfig["~/.config/openshell/
clusters, mTLS certs,
active_cluster"] + LocalConfig["~/.config/openshell/
gateways, mTLS certs,
active_gateway"] end %% ============================================================ - %% KUBERNETES CLUSTER (single Docker container) + %% GATEWAY AND COMPUTE PLATFORM %% ============================================================ - subgraph Cluster["OpenShell Cluster Container (Docker)"] + subgraph Cluster["Gateway and Compute Platform"] - subgraph K3s["k3s (v1.35.2-k3s1)"] - KubeAPI["Kubernetes API
:6443"] - HelmController["Helm Controller"] - LocalPathProv["local-path-provisioner"] - end + ComputeDriver["Compute Driver
(Docker, Podman,
Kubernetes, VM)"] + DockerAPI["Docker API"] + PodmanAPI["Podman API"] + KubeAPI["Kubernetes API
(optional)"] + VMDriver["VM Driver
(experimental)"] - subgraph NSNamespace["openshell namespace"] + subgraph NSNamespace["Gateway runtime"] - subgraph GatewayPod["Gateway StatefulSet"] + subgraph GatewayPod["Gateway Process"] Gateway["openshell-server
:8080
(gRPC + HTTP, mTLS)"] SQLite[("SQLite DB
/var/openshell/
openshell.db")] - SandboxWatcher["Sandbox Watcher"] - KubeEventTailer["Kube Event Tailer"] + SupRegistry["SupervisorSessionRegistry
(live sessions + pending relays)"] WatchBus["SandboxWatchBus
(in-memory broadcast)"] LogBus["TracingLogBus
(in-memory broadcast)"] end - subgraph SandboxPod["Sandbox Pod (1 per sandbox)"] + subgraph SandboxPod["Sandbox Workload
(container, pod, or VM)"] subgraph Supervisor["Sandbox Supervisor
(privileged user)"] - SSHServer["Embedded SSH
Server (russh)
:2222"] + SSHServer["Embedded SSH
Server (russh)
Unix socket
/run/openshell/ssh.sock"] + RelayBridge["Relay Bridge
(ConnectSupervisor +
RelayStream client)"] Proxy["HTTP CONNECT
Proxy
10.200.0.1:3128"] OPA["OPA Policy Engine
(regorus, in-process)"] InferenceRouter["Inference Router
(openshell-router)"] @@ -54,7 +54,7 @@ graph TB end end - subgraph ASNamespace["agent-sandbox-system namespace"] + subgraph ASNamespace["Kubernetes driver only"] CRDController["Agent Sandbox
CRD Controller"] end @@ -91,7 +91,7 @@ graph TB %% ============================================================ %% CONNECTIONS: User Machine --> Cluster %% ============================================================ - CLI -- "gRPC over HTTPS (mTLS)
:30051 NodePort" --> Gateway + CLI -- "gRPC over HTTPS (mTLS)
service / ingress / port-forward" --> Gateway TUI -- "gRPC polling (mTLS)
every 2s" --> Gateway SDK -- "gRPC over HTTPS (mTLS)" --> Gateway CLI -- "HTTP CONNECT upgrade
/connect/ssh (mTLS)" --> Gateway @@ -101,10 +101,20 @@ graph TB %% CONNECTIONS: Gateway internals %% ============================================================ Gateway --> SQLite - Gateway -- "Watch + CRUD
Sandbox CRDs" --> KubeAPI - SandboxWatcher -- "status changes" --> WatchBus - KubeEventTailer -- "K8s events" --> Gateway - Gateway -- "NSSH1 handshake
(HMAC-SHA256) + SSH
:2222" --> SSHServer + Gateway --> SupRegistry + Gateway -- "Create / delete / watch
sandboxes" --> ComputeDriver + ComputeDriver --> DockerAPI + ComputeDriver --> PodmanAPI + ComputeDriver --> KubeAPI + ComputeDriver --> VMDriver + ComputeDriver -- "status, platform events" --> Gateway + + %% ============================================================ + %% CONNECTIONS: Supervisor session (inbound from sandbox) + %% ============================================================ + RelayBridge -- "ConnectSupervisor
(persistent bidi stream)" --> SupRegistry + RelayBridge -- "RelayStream
(per-invocation byte bridge,
same HTTP/2 connection)" --> SupRegistry + RelayBridge -- "Unix socket
SSH bytes" --> SSHServer %% ============================================================ %% CONNECTIONS: CRD Controller @@ -123,7 +133,7 @@ graph TB %% ============================================================ %% CONNECTIONS: Sandbox --> Gateway (control plane) %% ============================================================ - Supervisor -- "gRPC (mTLS):
GetSandboxSettings
(policy + settings),
GetProviderEnvironment,
GetInferenceBundle,
PushSandboxLogs" --> Gateway + Supervisor -- "gRPC (mTLS):
GetSandboxConfig
(policy + settings),
GetProviderEnvironment,
GetInferenceBundle,
PushSandboxLogs" --> Gateway %% ============================================================ %% CONNECTIONS: Sandbox --> External (via proxy) @@ -140,14 +150,14 @@ graph TB InferenceRouter -- "HTTPS" --> NVIDIA_API %% ============================================================ - %% CONNECTIONS: Cluster bootstrap + %% CONNECTIONS: Image pulls %% ============================================================ - K3s -- "pulls images
at runtime" --> GHCR + ComputeDriver -- "pulls or schedules workloads
that pull images" --> GHCR %% ============================================================ - %% FILE SYNC + %% CLIENT SSH / EXEC (bytes tunneled via supervisor relay) %% ============================================================ - CLI -- "tar-over-SSH
(file sync)" --> SSHServer + CLI -- "HTTP CONNECT /connect/ssh
+ tar-over-SSH file sync
(bytes bridged through
SupervisorSessionRegistry)" --> Gateway %% ============================================================ %% STYLES @@ -164,12 +174,12 @@ graph TB classDef config fill:#90A4AE,stroke:#607D8B,color:#fff class CLI,TUI,SDK userComponent - class Gateway,SandboxWatcher,KubeEventTailer,WatchBus,LogBus gateway - class SSHServer,Proxy,OPA,InferenceRouter,CertCache sandbox + class Gateway,SupRegistry,WatchBus,LogBus gateway + class SSHServer,RelayBridge,Proxy,OPA,InferenceRouter,CertCache sandbox class Agent,Landlock,Seccomp,NetNS agent class SQLite datastore class Anthropic,OpenAI,NVIDIA_API,GitHub,GitLab,PyPI,NPM,LMStudio,VLLM,GHCR external - class KubeAPI,HelmController,LocalPathProv,CRDController k8s + class ComputeDriver,DockerAPI,PodmanAPI,KubeAPI,VMDriver,CRDController k8s class LocalConfig config ``` @@ -182,19 +192,21 @@ graph TB | Green | Sandbox supervisor | SSH server, HTTP CONNECT proxy, OPA engine, inference router | | Purple | Agent process & isolation | AI agent, Landlock, Seccomp, network namespace | | Indigo | Data stores | SQLite database | -| Dark blue | Kubernetes infrastructure | K8s API, Helm controller, CRD controller | +| Dark blue | Compute infrastructure | Docker API, Podman API, K8s API, VM driver | | Gray | External systems | AI APIs, code hosting, package registries, inference backends | ## Key Communication Flows -1. **CLI/SDK to Gateway**: All control-plane traffic uses gRPC over HTTPS with mutual TLS (mTLS). Single multiplexed port (8080 inside cluster, 30051 NodePort). +1. **CLI/SDK to Gateway**: Control-plane traffic uses gRPC over HTTPS with mutual TLS (mTLS) unless the gateway is explicitly deployed in plaintext mode behind a trusted transport. The gateway listens on one multiplexed service port. + +2. **Supervisor Session (inbound from sandbox)**: Each sandbox supervisor opens a persistent `ConnectSupervisor` bidi gRPC stream to the gateway over mTLS. The gateway tracks these in `SupervisorSessionRegistry`. When SSH or exec access is needed, the gateway sends `RelayOpen { channel_id }` on that stream; the supervisor responds by initiating a `RelayStream` RPC on the same HTTP/2 connection whose first frame is a `RelayInit { channel_id }`. Subsequent frames carry raw bytes in both directions. The gateway never dials the sandbox pod. -2. **SSH Access**: CLI connects via HTTP CONNECT upgrade at `/connect/ssh`. Gateway authenticates with session token, then bridges to sandbox SSH (port 2222) using NSSH1 HMAC-SHA256 handshake. +3. **SSH / Exec Access**: CLI connects via HTTP CONNECT upgrade at `/connect/ssh` (or calls `ExecSandbox` gRPC). The gateway authenticates, calls `open_relay`, and bridges the client bytes through the supervisor's `RelayStream` to the supervisor's in-sandbox SSH daemon, which binds to a Unix socket (`/run/openshell/ssh.sock`) rather than a TCP port. -3. **File Sync**: tar archives streamed over the SSH tunnel (no rsync dependency). +4. **File Sync**: tar archives streamed over the relay-tunneled SSH session (no rsync dependency). -4. **Sandbox to External**: All agent outbound traffic is forced through the HTTP CONNECT proxy (10.200.0.1:3128) via a network namespace veth pair. OPA/Rego policies evaluate every connection. TLS is automatically detected and terminated for credential injection; endpoints with `protocol` configured also get L7 request-level inspection. +5. **Sandbox to External**: All agent outbound traffic is forced through the HTTP CONNECT proxy (10.200.0.1:3128) via a network namespace veth pair. OPA/Rego policies evaluate every connection. TLS is automatically detected and terminated for credential injection; endpoints with `protocol` configured also get L7 request-level inspection. -5. **Inference Routing**: Inference requests are handled inside the sandbox by the openshell-router (not through the gateway). The gateway provides route configuration and credentials via gRPC; the sandbox executes HTTP requests directly to inference backends. +6. **Inference Routing**: Inference requests are handled inside the sandbox by the openshell-router (not through the gateway). The gateway provides route configuration and credentials via gRPC; the sandbox executes HTTP requests directly to inference backends. -6. **Sandbox to Gateway**: The sandbox supervisor uses gRPC (mTLS) to fetch policies and runtime settings (via `GetSandboxSettings`), provider credentials, inference bundles, and to push logs back to the gateway. The settings channel delivers typed key-value pairs alongside policy through a unified poll loop. +7. **Sandbox to Gateway (control plane)**: The sandbox supervisor uses gRPC (mTLS) to fetch policies and runtime settings (via `GetSandboxConfig`), provider credentials, inference bundles, and to push logs back to the gateway. The settings channel delivers typed key-value pairs alongside policy through a unified poll loop. This reuses the same mTLS connection that carries `ConnectSupervisor`. diff --git a/architecture/tui.md b/architecture/tui.md index 1a83e96d1a..850cebc886 100644 --- a/architecture/tui.md +++ b/architecture/tui.md @@ -1,10 +1,10 @@ # OpenShell TUI -The OpenShell TUI is a terminal user interface for OpenShell, inspired by [k9s](https://k9scli.io/). Instead of typing individual CLI commands to check cluster health, list sandboxes, and manage resources, the TUI gives you a real-time, keyboard-driven dashboard — everything updates automatically and you navigate with a few keystrokes. +The OpenShell TUI is a terminal user interface for OpenShell, inspired by [k9s](https://k9scli.io/). Instead of typing individual CLI commands to check gateway health, list sandboxes, and manage resources, the TUI gives you a real-time, keyboard-driven dashboard — everything updates automatically and you navigate with a few keystrokes. ## Launching the TUI -The TUI is a subcommand of the OpenShell CLI, so it inherits all your existing configuration — cluster selection, TLS settings, and verbosity flags all work the same way. +The TUI is a subcommand of the OpenShell CLI, so it inherits all your existing configuration — gateway selection, TLS settings, and verbosity flags all work the same way. ```bash openshell term # launch against the active gateway @@ -25,9 +25,9 @@ No separate configuration files or authentication are needed. The TUI divides the terminal into four horizontal regions: -``` +```text ┌─────────────────────────────────────────────────────────────────┐ -│ OpenShell ─ my-cluster ─ Dashboard ● Healthy │ ← title bar +│ OpenShell ─ my-gateway ─ Dashboard ● Healthy │ ← title bar ├─────────────────────────────────────────────────────────────────┤ │ │ │ (view content — Dashboard or Sandboxes) │ ← main area @@ -39,7 +39,7 @@ The TUI divides the terminal into four horizontal regions: └─────────────────────────────────────────────────────────────────┘ ``` -- **Title bar** — shows the OpenShell logo, cluster name, current view, and live cluster health status. +- **Title bar** — shows the OpenShell logo, gateway name, current view, and live gateway health status. - **Main area** — the active view (Dashboard or Sandboxes). - **Navigation bar** — lists available views with their shortcut keys, plus Help and Quit. - **Command bar** — appears when you press `:` to type a command (like vim). @@ -48,20 +48,21 @@ The TUI divides the terminal into four horizontal regions: ### Dashboard (press `1`) -The Dashboard is the home screen. It shows your cluster at a glance. +The Dashboard is the home screen. It shows your gateway at a glance. The dashboard is divided into a top info pane and a middle pane with two tabs: -- **Top pane**: Cluster name, gateway endpoint, health status, sandbox count. +- **Top pane**: Gateway name, gateway endpoint, health status, sandbox count. - **Middle pane**: Tabbed view toggled with `Tab`: - - **Providers** — provider configurations attached to the cluster. + - **Providers** — provider configurations attached to the gateway. - **Global Settings** — gateway-global runtime settings (fetched via `GetGatewaySettings`). **Health status** indicators: - - `●` **Healthy** (green) — everything is running normally. - - `◐` **Degraded** (yellow) — the cluster is up but something needs attention. - - `○` **Unhealthy** (red) — the cluster is not operating correctly. - - `…` — still connecting or status unknown. + +- `●` **Healthy** (green) — everything is running normally. +- `◐` **Degraded** (yellow) — the gateway is up but something needs attention. +- `○` **Unhealthy** (red) — the gateway is not operating correctly. +- `…` — still connecting or status unknown. **Global policy indicator**: When a global policy is active, the gateway row shows `Global Policy Active (vN)` in yellow (the `status_warn` style). The TUI detects this by polling `ListSandboxPolicies` with `global: true, limit: 1` on each tick and checking if the latest revision has `PolicyStatus::Loaded`. See `crates/openshell-tui/src/ui/dashboard.rs`. @@ -80,7 +81,7 @@ Both edit and delete operations display a confirmation modal before applying. Ch ### Sandboxes (press `2`) -The Sandboxes view shows a table of all sandboxes in the cluster: +The Sandboxes view shows a table of all sandboxes in the gateway: | Column | Description | |--------|-------------| @@ -149,7 +150,7 @@ Press `Esc` to cancel and return to Normal mode. `Backspace` deletes characters ## Data Refresh -The TUI automatically polls the cluster every **2 seconds**. Cluster health, the sandbox list, and global settings all update on each tick, so the display stays current without manual refreshing. This uses the same gRPC calls as the CLI — no additional server-side setup is required. +The TUI automatically polls the gateway every **2 seconds**. Gateway health, the sandbox list, and global settings all update on each tick, so the display stays current without manual refreshing. This uses the same gRPC calls as the CLI — no additional server-side setup is required. When viewing a sandbox, the policy pane auto-refreshes when a new policy version is detected. The sandbox list response includes `current_policy_version` for each sandbox; on every tick the TUI compares this against the currently displayed policy version and re-fetches the full policy only when they differ. This avoids extra RPCs during normal operation while ensuring policy updates appear within the polling interval. The user's scroll position is preserved across auto-refreshes. @@ -174,6 +175,7 @@ The TUI supports creating sandboxes with port forwarding directly from the creat Forwarded ports are displayed in the **NOTES** column of the sandbox table as `fwd:8080,3000` and in the **Forwards** row of the sandbox detail view. Port forwarding lifecycle: + - **On create**: The TUI polls for sandbox readiness (up to 30 attempts at 2-second intervals), then spawns SSH tunnels. - **On delete**: Any active forwards for the sandbox are automatically stopped before deletion. - **PID tracking**: Forward PIDs are stored in `~/.config/openshell/forwards/-.pid`, shared with the CLI. diff --git a/crates/openshell-bootstrap/src/build.rs b/crates/openshell-bootstrap/src/build.rs index eaa2213118..9caeca57aa 100644 --- a/crates/openshell-bootstrap/src/build.rs +++ b/crates/openshell-bootstrap/src/build.rs @@ -1,52 +1,72 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Build and push container images into a k3s gateway. +//! Build container images for gateway runtimes. //! //! This module wraps bollard's `build_image()` API to build a container image -//! from a Dockerfile and build context, then reuses the existing push pipeline -//! to import the image into the gateway's containerd runtime. +//! from a Dockerfile and build context. Kubernetes deployments reuse the +//! existing push pipeline to import the image into the gateway's containerd +//! runtime. VM deployments keep the built image in the local Docker daemon and +//! pass an internal local-image reference to the VM driver. use std::collections::HashMap; use std::path::Path; +use std::time::Duration; use bollard::Docker; use bollard::query_parameters::BuildImageOptionsBuilder; use futures::StreamExt; use miette::{IntoDiagnostic, Result, WrapErr}; +use tokio::time::timeout; use crate::constants::container_name; use crate::push::push_local_images; -/// Build a container image from a Dockerfile and push it into the gateway. +/// Maximum gap between Docker build stream events before a build is treated +/// as stuck. /// -/// This is used by `openshell sandbox create --from `. It: -/// 1. Creates a tar archive of the build context directory. -/// 2. Sends it to the local Docker daemon via `build_image()`. -/// 3. Pushes the resulting image into the gateway's containerd via the -/// existing `push_local_images()` pipeline. +/// Total silence longer than this on under-provisioned container runtimes +/// (e.g. default Colima 2 vCPU / 2 GiB on macOS) reliably indicates a +/// deadlocked builder that will never recover. The default leaves headroom +/// for legitimately quiet steps (a single long `RUN` that produces no output) +/// — override with `OPENSHELL_BUILD_NO_PROGRESS_TIMEOUT_SECS` if a specific +/// build needs more time, or shorter for CI tightening. +const DEFAULT_BUILD_NO_PROGRESS_TIMEOUT_SECS: u64 = 1800; + +/// Build a container image from a Dockerfile using the local Docker daemon. +/// +/// This is used by `openshell sandbox create --from ` for both the +/// Kubernetes and VM backends. The image remains available in the local Docker +/// daemon so the caller can either hand the resulting tag directly to the VM +/// backend or import it into a local gateway containerd runtime. #[allow(clippy::implicit_hasher)] -pub async fn build_and_push_image( +pub async fn build_local_image( dockerfile_path: &Path, tag: &str, context_dir: &Path, - gateway_name: &str, build_args: &HashMap, on_log: &mut impl FnMut(String), ) -> Result<()> { - // 1. Build the image locally. on_log(format!( "Building image {tag} from {}", dockerfile_path.display() )); build_image(dockerfile_path, tag, context_dir, build_args, on_log).await?; on_log(format!("Built image {tag}")); + Ok(()) +} - // 2. Push into the gateway. +/// Push a locally-built image into the gateway's containerd runtime. +#[allow(clippy::implicit_hasher)] +pub async fn push_image_into_gateway( + tag: &str, + gateway_name: &str, + on_log: &mut impl FnMut(String), +) -> Result<()> { on_log(format!( "Pushing image {tag} into gateway \"{gateway_name}\"" )); - let local_docker = Docker::connect_with_local_defaults() + let local_docker = crate::docker::connect_local_for_large_transfers() .into_diagnostic() .wrap_err("failed to connect to local Docker daemon")?; let container = container_name(gateway_name); @@ -57,6 +77,28 @@ pub async fn build_and_push_image( Ok(()) } +/// Build a container image from a Dockerfile and push it into the gateway. +/// +/// This is used by `openshell sandbox create --from ` when the +/// active gateway is the local Kubernetes deployment. It: +/// 1. Creates a tar archive of the build context directory. +/// 2. Sends it to the local Docker daemon via `build_image()`. +/// 3. Pushes the resulting image into the gateway's containerd via the +/// existing `push_local_images()` pipeline. +#[allow(clippy::implicit_hasher)] +pub async fn build_and_push_image( + dockerfile_path: &Path, + tag: &str, + context_dir: &Path, + gateway_name: &str, + build_args: &HashMap, + on_log: &mut impl FnMut(String), +) -> Result<()> { + build_local_image(dockerfile_path, tag, context_dir, build_args, on_log).await?; + push_image_into_gateway(tag, gateway_name, on_log).await?; + Ok(()) +} + /// Build a container image using the local Docker daemon. /// /// Creates a tar archive of `context_dir`, sends it to Docker with the @@ -97,9 +139,30 @@ async fn build_image( let body = bollard::body_full(bytes::Bytes::from(context_tar)); let mut stream = docker.build_image(options, None, Some(body)); + let no_progress_secs: u64 = std::env::var("OPENSHELL_BUILD_NO_PROGRESS_TIMEOUT_SECS") + .ok() + .and_then(|s| s.parse().ok()) + .filter(|&n| n > 0) + .unwrap_or(DEFAULT_BUILD_NO_PROGRESS_TIMEOUT_SECS); + let no_progress_timeout = Duration::from_secs(no_progress_secs); + + loop { + let next = match timeout(no_progress_timeout, stream.next()).await { + Ok(Some(result)) => result, + Ok(None) => break, + Err(_) => { + return Err(miette::miette!( + "Docker build produced no output for {}s. This usually means the container \ + runtime is under-provisioned (CPU/memory) and the builder has deadlocked; \ + check `docker info` (NCPU, MemTotal) and increase Colima/Docker Desktop \ + resources before retrying. If a legitimate build step is just quiet, raise \ + the threshold with OPENSHELL_BUILD_NO_PROGRESS_TIMEOUT_SECS=.", + no_progress_timeout.as_secs() + )); + } + }; - while let Some(result) = stream.next().await { - let info = result + let info = next .into_diagnostic() .wrap_err("Docker build stream error")?; diff --git a/crates/openshell-bootstrap/src/docker.rs b/crates/openshell-bootstrap/src/docker.rs index be086e534e..c18c938aaf 100644 --- a/crates/openshell-bootstrap/src/docker.rs +++ b/crates/openshell-bootstrap/src/docker.rs @@ -23,6 +23,24 @@ use std::collections::HashMap; const REGISTRY_NAMESPACE_DEFAULT: &str = "openshell"; +/// Default total HTTP timeout for Docker API calls that stream large payloads +/// (e.g. `docker save` used by `sandbox create --from`). Bollard's own +/// `connect_with_local_defaults()` ceiling is 120s, which is far too short for +/// multi-GB image exports — a 7 GB image on a laptop SSD takes ~4–5 minutes. +/// One hour is a safe upper bound; override with `OPENSHELL_DOCKER_TIMEOUT_SECS`. +pub const DEFAULT_LARGE_TRANSFER_TIMEOUT_SECS: u64 = 3600; + +/// Build a local-Docker client suitable for large streaming transfers. +/// Respects `OPENSHELL_DOCKER_TIMEOUT_SECS` (in seconds); falls back to +/// [`DEFAULT_LARGE_TRANSFER_TIMEOUT_SECS`] when unset or unparseable. +pub fn connect_local_for_large_transfers() -> std::result::Result { + let secs: u64 = std::env::var("OPENSHELL_DOCKER_TIMEOUT_SECS") + .ok() + .and_then(|s| s.parse().ok()) + .unwrap_or(DEFAULT_LARGE_TRANSFER_TIMEOUT_SECS); + Ok(Docker::connect_with_local_defaults()?.with_timeout(std::time::Duration::from_secs(secs))) +} + /// Resolve the raw GPU device-ID list, replacing the `"auto"` sentinel with a /// concrete device ID based on whether CDI is enabled on the daemon. /// @@ -32,7 +50,7 @@ const REGISTRY_NAMESPACE_DEFAULT: &str = "openshell"; /// | `["legacy"]` | `["legacy"]` — pass through to the non-CDI fallback path | /// | `["auto"]` | `["nvidia.com/gpu=all"]` if CDI enabled, else `["legacy"]` | /// | `[cdi-ids…]` | unchanged | -pub(crate) fn resolve_gpu_device_ids(gpu: &[String], cdi_enabled: bool) -> Vec { +pub fn resolve_gpu_device_ids(gpu: &[String], cdi_enabled: bool) -> Vec { match gpu { [] => vec![], [v] if v == "auto" => { @@ -328,22 +346,25 @@ pub async fn find_gateway_container(docker: &Docker, port: Option) -> Resul let matches: Vec = containers .iter() - .filter(|c| is_gateway_image(c) && port.map_or(true, |p| has_port(c, p))) + .filter(|c| is_gateway_image(c) && port.is_none_or(|p| has_port(c, p))) .filter_map(container_name) .collect(); match matches.len() { 0 => { - let hint = if let Some(p) = port { - format!( - "No openshell gateway container found listening on port {p}.\n\ + let hint = port.map_or_else( + || { + "No openshell gateway container found.\n\ Is the gateway running? Check with: docker ps" - ) - } else { - "No openshell gateway container found.\n\ - Is the gateway running? Check with: docker ps" - .to_string() - }; + .to_string() + }, + |p| { + format!( + "No openshell gateway container found listening on port {p}.\n\ + Is the gateway running? Check with: docker ps" + ) + }, + ); Err(miette::miette!("{hint}")) } 1 => Ok(matches.into_iter().next().unwrap()), @@ -470,6 +491,8 @@ pub async fn ensure_image( /// Returns the actual host port the container is using. When an existing /// container is reused (same image), this may differ from `gateway_port` /// because the container was originally created with a different port. +// Refactoring this signature would touch many call sites across the workspace. +#[allow(clippy::too_many_arguments)] pub async fn ensure_container( docker: &Docker, name: &str, @@ -483,6 +506,12 @@ pub async fn ensure_container( registry_token: Option<&str>, device_ids: &[String], resume: bool, + oidc_issuer: Option<&str>, + oidc_audience: &str, + oidc_roles_claim: Option<&str>, + oidc_admin_role: Option<&str>, + oidc_user_role: Option<&str>, + oidc_scopes_claim: Option<&str>, ) -> Result { let container_name = container_name(name); @@ -726,10 +755,7 @@ pub async fn ensure_container( // When OPENSHELL_PUSH_IMAGES is set the entrypoint overrides the baked-in // HelmChart manifest so k3s uses the locally-pushed images with // IfNotPresent pull policy instead of pulling from the remote registry. - let push_mode = std::env::var("OPENSHELL_PUSH_IMAGES") - .ok() - .filter(|v| !v.trim().is_empty()) - .is_some(); + let push_mode = std::env::var("OPENSHELL_PUSH_IMAGES").is_ok_and(|v| !v.trim().is_empty()); let effective_tag = std::env::var("IMAGE_TAG") .ok() .filter(|v| !v.trim().is_empty()) @@ -764,6 +790,25 @@ pub async fn ensure_container( env_vars.push("GPU_ENABLED=true".to_string()); } + // OIDC JWT authentication: pass issuer and audience to the entrypoint + // so the HelmChart manifest configures the server pod for JWT validation. + if let Some(issuer) = oidc_issuer { + env_vars.push(format!("OIDC_ISSUER={issuer}")); + env_vars.push(format!("OIDC_AUDIENCE={oidc_audience}")); + if let Some(claim) = oidc_roles_claim { + env_vars.push(format!("OIDC_ROLES_CLAIM={claim}")); + } + if let Some(role) = oidc_admin_role { + env_vars.push(format!("OIDC_ADMIN_ROLE={role}")); + } + if let Some(role) = oidc_user_role { + env_vars.push(format!("OIDC_USER_ROLE={role}")); + } + if let Some(claim) = oidc_scopes_claim { + env_vars.push(format!("OIDC_SCOPES_CLAIM={claim}")); + } + } + let env = Some(env_vars); let config = ContainerCreateBody { @@ -845,22 +890,22 @@ pub async fn check_port_conflicts( let ports = container.ports.as_deref().unwrap_or_default(); for port in ports { - if let Some(public) = port.public_port { - if needed_ports.contains(&public) { - let cname = names - .first() - .map(|n| n.trim_start_matches('/').to_string()) - .unwrap_or_else(|| { - container - .id - .clone() - .unwrap_or_else(|| "".to_string()) - }); - conflicts.push(PortConflict { - container_name: cname, - host_port: public, - }); - } + if let Some(public) = port.public_port + && needed_ports.contains(&public) + { + let cname = names.first().map_or_else( + || { + container + .id + .clone() + .unwrap_or_else(|| "".to_string()) + }, + |n| n.trim_start_matches('/').to_string(), + ); + conflicts.push(PortConflict { + container_name: cname, + host_port: public, + }); } } } @@ -1353,6 +1398,9 @@ mod tests { ); } + // Test-only: mutates DOCKER_HOST env var via std::env::set_var/remove_var, + // which require unsafe in the 2024 edition. + #[allow(unsafe_code)] #[test] fn docker_not_reachable_error_with_docker_host() { // Simulate: DOCKER_HOST is set but daemon unresponsive. diff --git a/crates/openshell-bootstrap/src/errors.rs b/crates/openshell-bootstrap/src/errors.rs index 9e385c680e..8a6ccf5bdd 100644 --- a/crates/openshell-bootstrap/src/errors.rs +++ b/crates/openshell-bootstrap/src/errors.rs @@ -375,7 +375,7 @@ fn diagnose_missing_supervisor(gateway_name: &str) -> GatewayFailureDiagnosis { was not found in the gateway container. This binary is side-loaded into every \ sandbox pod via a hostPath volume mount. Without it, all sandbox pods will \ crash immediately with \"no such file or directory\". This typically means the \ - cluster image was built or published without the supervisor-builder stage." + cluster image was built or published without the staged prebuilt openshell-sandbox binary." .to_string(), recovery_steps: vec![ RecoveryStep::with_command( diff --git a/crates/openshell-bootstrap/src/lib.rs b/crates/openshell-bootstrap/src/lib.rs index 71d223d662..9651d367f5 100644 --- a/crates/openshell-bootstrap/src/lib.rs +++ b/crates/openshell-bootstrap/src/lib.rs @@ -5,6 +5,7 @@ pub mod build; pub mod edge_token; pub mod errors; pub mod image; +pub mod oidc_token; pub mod constants; mod docker; @@ -117,12 +118,26 @@ pub struct DeployOptions { /// - `[]` — no GPU passthrough (default) /// - `["legacy"]` — internal non-CDI fallback path (`driver="nvidia"`, `count=-1`) /// - `["auto"]` — resolved at deploy time: CDI if enabled on the daemon, else the non-CDI fallback - /// - `[cdi-ids…]` — CDI DeviceRequest with the given device IDs + /// - `[cdi-ids…]` — CDI `DeviceRequest` with the given device IDs pub gpu: Vec, /// When true, destroy any existing gateway resources before deploying. /// When false, an existing gateway is left as-is and deployment is /// skipped (the caller is responsible for prompting the user first). pub recreate: bool, + /// OIDC issuer URL. When set, the server validates Bearer JWTs. + pub oidc_issuer: Option, + /// OIDC audience for the API resource server. Defaults to "openshell-cli". + pub oidc_audience: String, + /// OIDC client ID for CLI login. Defaults to "openshell-cli". + pub oidc_client_id: String, + /// OIDC roles claim path (e.g. `realm_access.roles`). + pub oidc_roles_claim: Option, + /// OIDC admin role name. + pub oidc_admin_role: Option, + /// OIDC user role name. + pub oidc_user_role: Option, + /// OIDC scopes claim path. When set, the server enforces scope-based permissions. + pub oidc_scopes_claim: Option, } impl DeployOptions { @@ -139,6 +154,13 @@ impl DeployOptions { registry_token: None, gpu: vec![], recreate: false, + oidc_issuer: None, + oidc_audience: "openshell-cli".to_string(), + oidc_client_id: "openshell-cli".to_string(), + oidc_roles_claim: None, + oidc_admin_role: None, + oidc_user_role: None, + oidc_scopes_claim: None, } } @@ -208,6 +230,48 @@ impl DeployOptions { self.recreate = recreate; self } + + /// Set the OIDC issuer URL for JWT-based authentication. + #[must_use] + pub fn with_oidc_issuer(mut self, issuer: impl Into) -> Self { + self.oidc_issuer = Some(issuer.into()); + self + } + + /// Set the OIDC audience (client ID). + #[must_use] + pub fn with_oidc_audience(mut self, audience: impl Into) -> Self { + self.oidc_audience = audience.into(); + self + } +} + +fn apply_oidc_gateway_metadata( + metadata: &mut GatewayMetadata, + resume: bool, + existing: Option<&GatewayMetadata>, + oidc_issuer: Option<&str>, + oidc_client_id: &str, + oidc_audience: &str, +) { + if let Some(issuer) = oidc_issuer { + metadata.auth_mode = Some("oidc".to_string()); + metadata.oidc_issuer = Some(issuer.to_string()); + metadata.oidc_client_id = Some(oidc_client_id.to_string()); + metadata.oidc_audience = Some(oidc_audience.to_string()); + return; + } + + if resume + && let Some(existing) = existing + && existing.auth_mode.as_deref() == Some("oidc") + { + metadata.auth_mode.clone_from(&existing.auth_mode); + metadata.oidc_issuer.clone_from(&existing.oidc_issuer); + metadata.oidc_client_id.clone_from(&existing.oidc_client_id); + metadata.oidc_audience.clone_from(&existing.oidc_audience); + metadata.oidc_scopes.clone_from(&existing.oidc_scopes); + } } #[derive(Debug, Clone)] @@ -272,6 +336,13 @@ where let registry_token = options.registry_token; let gpu = options.gpu; let recreate = options.recreate; + let oidc_issuer = options.oidc_issuer; + let oidc_audience = options.oidc_audience; + let oidc_client_id = options.oidc_client_id; + let oidc_roles_claim = options.oidc_roles_claim; + let oidc_admin_role = options.oidc_admin_role; + let oidc_user_role = options.oidc_user_role; + let oidc_scopes_claim = options.oidc_scopes_claim; // Wrap on_log in Arc> so we can share it with pull_remote_image // which needs a 'static callback for the bollard streaming pull. @@ -340,8 +411,8 @@ where // the image to recreate the container, so the pull must happen. let need_image = !resume || !resume_container_exists; if need_image { + log("[status] Downloading gateway".to_string()); if remote_opts.is_some() { - log("[status] Downloading gateway".to_string()); let on_log_clone = Arc::clone(&on_log); let progress_cb = move |msg: String| { if let Ok(mut f) = on_log_clone.lock() { @@ -358,7 +429,6 @@ where .await?; } else { // Local deployment: ensure image exists (pull if needed) - log("[status] Downloading gateway".to_string()); ensure_image( &target_docker, &image_ref, @@ -458,6 +528,12 @@ where registry_token.as_deref(), &device_ids, resume, + oidc_issuer.as_deref(), + &oidc_audience, + oidc_roles_claim.as_deref(), + oidc_admin_role.as_deref(), + oidc_user_role.as_deref(), + oidc_scopes_claim.as_deref(), ) .await?; let port = actual_port; @@ -521,7 +597,10 @@ where .collect(); if !images.is_empty() { log("[status] Deploying components".to_string()); - let local_docker = Docker::connect_with_local_defaults().into_diagnostic()?; + // Long-timeout client: `docker save` of multi-GB component + // images streams past bollard's 120s default. See + // docker::connect_local_for_large_transfers(). + let local_docker = docker::connect_local_for_large_transfers().into_diagnostic()?; let container = container_name(&name); let on_log_ref = Arc::clone(&on_log); let mut push_log = move |msg: String| { @@ -555,14 +634,29 @@ where wait_for_gateway_ready(&target_docker, &name, &mut gateway_log).await?; } - // Create and store gateway metadata. - let metadata = create_gateway_metadata_with_host( + // Create and store gateway metadata. On resume, preserve existing + // OIDC fields so a bare `gateway start` without `--oidc-*` flags + // doesn't erase a previously configured OIDC registration. + let mut metadata = create_gateway_metadata_with_host( &name, remote_opts.as_ref(), port, ssh_gateway_host.as_deref(), disable_tls, ); + let existing_metadata = if resume { + load_gateway_metadata(&name).ok() + } else { + None + }; + apply_oidc_gateway_metadata( + &mut metadata, + resume, + existing_metadata.as_ref(), + oidc_issuer.as_deref(), + &oidc_client_id, + &oidc_audience, + ); store_gateway_metadata(&name, &metadata)?; Ok(metadata) @@ -729,16 +823,16 @@ pub async fn gateway_container_logs( Ok(()) } -/// Fetch the last `n` lines of container logs for a local gateway as a -/// `String`. This is a convenience wrapper for diagnostic call sites (e.g. -/// failure diagnosis in the CLI) that do not hold a Docker client handle. +/// Fetch the last `n` lines of container logs for a local gateway as a `String`. +/// +/// This is a convenience wrapper for diagnostic call sites (e.g. failure +/// diagnosis in the CLI) that do not hold a Docker client handle. /// /// Returns an empty string on any Docker/connection error so callers don't /// need to worry about error handling. pub async fn fetch_gateway_logs(name: &str, n: usize) -> String { - let docker = match Docker::connect_with_local_defaults() { - Ok(d) => d, - Err(_) => return String::new(), + let Ok(docker) = Docker::connect_with_local_defaults() else { + return String::new(); }; let container = container_name(name); fetch_recent_logs(&docker, &container, n).await @@ -1095,7 +1189,9 @@ async fn wait_for_namespace( ) -> Result<()> { use miette::WrapErr; - let attempts = 60; + // Shared CPU runners can take several minutes to cold-start k3s, apply + // bundled manifests, and let the k3s Helm controller create the namespace. + let attempts = 150; let max_backoff = std::time::Duration::from_secs(2); let mut backoff = std::time::Duration::from_millis(200); @@ -1214,4 +1310,82 @@ mod tests { ); } } + + #[test] + fn apply_oidc_gateway_metadata_sets_explicit_values() { + let mut metadata = GatewayMetadata::default(); + apply_oidc_gateway_metadata( + &mut metadata, + false, + None, + Some("http://issuer.test/realm"), + "openshell-cli", + "openshell-api", + ); + + assert_eq!(metadata.auth_mode.as_deref(), Some("oidc")); + assert_eq!( + metadata.oidc_issuer.as_deref(), + Some("http://issuer.test/realm") + ); + assert_eq!(metadata.oidc_client_id.as_deref(), Some("openshell-cli")); + assert_eq!(metadata.oidc_audience.as_deref(), Some("openshell-api")); + } + + #[test] + fn apply_oidc_gateway_metadata_preserves_existing_oidc_on_resume() { + let mut metadata = GatewayMetadata::default(); + let existing = GatewayMetadata { + auth_mode: Some("oidc".to_string()), + oidc_issuer: Some("http://issuer.test/realm".to_string()), + oidc_client_id: Some("openshell-cli".to_string()), + oidc_audience: Some("openshell-api".to_string()), + oidc_scopes: Some("sandbox:read".to_string()), + ..GatewayMetadata::default() + }; + + apply_oidc_gateway_metadata( + &mut metadata, + true, + Some(&existing), + None, + "ignored-client", + "ignored-audience", + ); + + assert_eq!(metadata.auth_mode.as_deref(), Some("oidc")); + assert_eq!( + metadata.oidc_issuer.as_deref(), + Some("http://issuer.test/realm") + ); + assert_eq!(metadata.oidc_client_id.as_deref(), Some("openshell-cli")); + assert_eq!(metadata.oidc_audience.as_deref(), Some("openshell-api")); + assert_eq!(metadata.oidc_scopes.as_deref(), Some("sandbox:read")); + } + + #[test] + fn apply_oidc_gateway_metadata_does_not_preserve_without_resume() { + let mut metadata = GatewayMetadata::default(); + let existing = GatewayMetadata { + auth_mode: Some("oidc".to_string()), + oidc_issuer: Some("http://issuer.test/realm".to_string()), + oidc_client_id: Some("openshell-cli".to_string()), + oidc_audience: Some("openshell-api".to_string()), + ..GatewayMetadata::default() + }; + + apply_oidc_gateway_metadata( + &mut metadata, + false, + Some(&existing), + None, + "ignored-client", + "ignored-audience", + ); + + assert!(metadata.auth_mode.is_none()); + assert!(metadata.oidc_issuer.is_none()); + assert!(metadata.oidc_client_id.is_none()); + assert!(metadata.oidc_audience.is_none()); + } } diff --git a/crates/openshell-bootstrap/src/metadata.rs b/crates/openshell-bootstrap/src/metadata.rs index 8e6b8a0703..b1bb36351a 100644 --- a/crates/openshell-bootstrap/src/metadata.rs +++ b/crates/openshell-bootstrap/src/metadata.rs @@ -9,7 +9,7 @@ use serde::{Deserialize, Serialize}; use std::path::PathBuf; /// Gateway metadata stored alongside deployment info. -#[derive(Debug, Clone, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct GatewayMetadata { /// The gateway name. pub name: String, @@ -46,6 +46,41 @@ pub struct GatewayMetadata { alias = "cf_auth_url" )] pub edge_auth_url: Option, + + /// OIDC issuer URL (set when `auth_mode == "oidc"`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub oidc_issuer: Option, + + /// OIDC client ID for the CLI login flow (set when `auth_mode == "oidc"`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub oidc_client_id: Option, + + /// OIDC audience for the resource server (API). When different from + /// `client_id`, the CLI requests this audience in the token exchange. + /// When `None`, defaults to the `client_id`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub oidc_audience: Option, + + /// Space-separated `OAuth2` scopes to request during OIDC login. + /// When set, tokens will include these scopes for fine-grained access control. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub oidc_scopes: Option, + + /// Local VM driver state directory for standalone VM gateways. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub vm_driver_state_dir: Option, + + /// Whether the CLI manages this gateway's full lifecycle (deploy, + /// stop, destroy). + /// + /// - `Some(true)` — deployed via `gateway start`; destroy/stop operate on + /// the underlying container or VM. + /// - `Some(false)` — registered via `gateway add`; destroy/stop only remove + /// the local registration metadata. + /// - `None` — legacy metadata written before this field existed; the CLI + /// falls back to the previous heuristic (`is_remote`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub client_lifecycle_managed: Option, } impl GatewayMetadata { @@ -134,8 +169,8 @@ pub fn create_gateway_metadata_with_host( remote_host, resolved_host, auth_mode: disable_tls.then(|| "plaintext".to_string()), - edge_team_domain: None, - edge_auth_url: None, + client_lifecycle_managed: Some(true), + ..Default::default() } } @@ -305,12 +340,11 @@ pub fn load_last_sandbox(gateway: &str) -> Option { /// This should be called after a sandbox is deleted so that subsequent commands /// don't try to connect to a sandbox that no longer exists. pub fn clear_last_sandbox_if_matches(gateway: &str, sandbox: &str) { - if let Some(current) = load_last_sandbox(gateway) { - if current == sandbox { - if let Ok(path) = last_sandbox_path(gateway) { - let _ = std::fs::remove_file(path); - } - } + if let Some(current) = load_last_sandbox(gateway) + && current == sandbox + && let Ok(path) = last_sandbox_path(gateway) + { + let _ = std::fs::remove_file(path); } } @@ -461,9 +495,7 @@ mod tests { gateway_port: 8080, remote_host: Some("user@openshell-dev".to_string()), resolved_host: Some("10.0.0.5".to_string()), - auth_mode: None, - edge_team_domain: None, - edge_auth_url: None, + ..Default::default() }; let json = serde_json::to_string(&meta).unwrap(); let parsed: GatewayMetadata = serde_json::from_str(&json).unwrap(); @@ -487,6 +519,53 @@ mod tests { assert!(parsed.resolved_host.is_none()); } + #[test] + fn metadata_deserialize_without_client_lifecycle_managed_field() { + // Legacy metadata files won't have the client_lifecycle_managed field. + // Ensure backwards compatibility: defaults to None. + let json = r#"{ + "name": "test", + "gateway_endpoint": "https://127.0.0.1:8080", + "is_remote": false, + "gateway_port": 8080 + }"#; + let parsed: GatewayMetadata = serde_json::from_str(json).unwrap(); + assert_eq!(parsed.client_lifecycle_managed, None); + } + + #[test] + fn metadata_roundtrip_with_client_lifecycle_managed_field() { + let meta = GatewayMetadata { + name: "test".to_string(), + gateway_endpoint: "https://127.0.0.1:8080".to_string(), + gateway_port: 8080, + client_lifecycle_managed: Some(false), + ..Default::default() + }; + let json = serde_json::to_string(&meta).unwrap(); + assert!(json.contains(r#""client_lifecycle_managed":false"#)); + let parsed: GatewayMetadata = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.client_lifecycle_managed, Some(false)); + } + + #[test] + fn metadata_omits_client_lifecycle_managed_when_none() { + let meta = GatewayMetadata { + name: "test".to_string(), + gateway_endpoint: "https://127.0.0.1:8080".to_string(), + gateway_port: 8080, + ..Default::default() + }; + let json = serde_json::to_string(&meta).unwrap(); + assert!(!json.contains("client_lifecycle_managed")); + } + + #[test] + fn create_gateway_metadata_sets_client_lifecycle_managed_true() { + let meta = create_gateway_metadata("test", None, 8080); + assert_eq!(meta.client_lifecycle_managed, Some(true)); + } + #[test] fn local_gateway_metadata_with_gateway_host_override() { let meta = create_gateway_metadata_with_host( @@ -552,13 +631,8 @@ mod tests { let meta = GatewayMetadata { name: "t".into(), gateway_endpoint: "https://localhost:8080".into(), - is_remote: false, gateway_port: 8080, - remote_host: None, - resolved_host: None, - auth_mode: None, - edge_team_domain: None, - edge_auth_url: None, + ..Default::default() }; assert_eq!(meta.gateway_host(), None); } @@ -572,9 +646,7 @@ mod tests { gateway_port: 8080, remote_host: Some("user@10.0.0.5".into()), resolved_host: Some("10.0.0.5".into()), - auth_mode: None, - edge_team_domain: None, - edge_auth_url: None, + ..Default::default() }; assert_eq!(meta.gateway_host(), Some("10.0.0.5")); } diff --git a/crates/openshell-bootstrap/src/oidc_token.rs b/crates/openshell-bootstrap/src/oidc_token.rs new file mode 100644 index 0000000000..19c6cabaa6 --- /dev/null +++ b/crates/openshell-bootstrap/src/oidc_token.rs @@ -0,0 +1,92 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! OIDC token storage. +//! +//! Stores OIDC token bundles (access token, refresh token, metadata) at +//! `$XDG_CONFIG_HOME/openshell/gateways//oidc_token.json`. +//! File permissions are `0600` (owner-only). + +use crate::paths::gateways_dir; +use miette::{IntoDiagnostic, Result, WrapErr}; +use openshell_core::paths::{ensure_parent_dir_restricted, set_file_owner_only}; +use serde::{Deserialize, Serialize}; +use std::path::PathBuf; + +/// OIDC token bundle persisted to disk. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct OidcTokenBundle { + /// `OAuth2` access token (JWT). + pub access_token: String, + + /// `OAuth2` refresh token. `None` for `client_credentials` grants. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub refresh_token: Option, + + /// Unix timestamp when the access token expires. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub expires_at: Option, + + /// OIDC issuer URL. + pub issuer: String, + + /// OIDC client ID used to obtain the token. + pub client_id: String, +} + +/// Path to the stored OIDC token bundle for a gateway. +pub fn oidc_token_path(gateway_name: &str) -> Result { + Ok(gateways_dir()?.join(gateway_name).join("oidc_token.json")) +} + +/// Store an OIDC token bundle for a gateway. +pub fn store_oidc_token(gateway_name: &str, bundle: &OidcTokenBundle) -> Result<()> { + let path = oidc_token_path(gateway_name)?; + ensure_parent_dir_restricted(&path)?; + let json = serde_json::to_string_pretty(bundle) + .into_diagnostic() + .wrap_err("failed to serialize OIDC token bundle")?; + std::fs::write(&path, json) + .into_diagnostic() + .wrap_err_with(|| format!("failed to write OIDC token to {}", path.display()))?; + set_file_owner_only(&path)?; + Ok(()) +} + +/// Load a stored OIDC token bundle for a gateway. +/// +/// Returns `None` if the token file does not exist or cannot be parsed. +pub fn load_oidc_token(gateway_name: &str) -> Option { + let path = oidc_token_path(gateway_name).ok()?; + if !path.exists() { + return None; + } + let contents = std::fs::read_to_string(&path).ok()?; + serde_json::from_str(&contents).ok() +} + +/// Remove a stored OIDC token. +pub fn remove_oidc_token(gateway_name: &str) -> Result<()> { + let path = oidc_token_path(gateway_name)?; + if path.exists() { + std::fs::remove_file(&path) + .into_diagnostic() + .wrap_err_with(|| format!("failed to remove {}", path.display()))?; + } + Ok(()) +} + +/// Check if the stored access token is expired or near expiry. +/// +/// Returns `true` if the token expires within the next 30 seconds. +pub fn is_token_expired(bundle: &OidcTokenBundle) -> bool { + let Some(expires_at) = bundle.expires_at else { + // No expiry info — assume valid. + return false; + }; + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_secs(); + now + 30 >= expires_at +} diff --git a/crates/openshell-bootstrap/src/runtime.rs b/crates/openshell-bootstrap/src/runtime.rs index 0f9a96e6bb..5bbfb5b6c7 100644 --- a/crates/openshell-bootstrap/src/runtime.rs +++ b/crates/openshell-bootstrap/src/runtime.rs @@ -136,7 +136,7 @@ where The file /opt/openshell/bin/openshell-sandbox was not found in the gateway \ container. Without it, sandbox pods cannot start.\n\n\ This usually means the cluster image was built or published without the \ - supervisor-builder stage.\n\n\ + staged prebuilt openshell-sandbox binary.\n\n\ To fix:\n \ 1. Rebuild the cluster image: mise run docker:build:cluster\n \ 2. Or update to a cluster image that includes the supervisor binary\n \ diff --git a/crates/openshell-cli/Cargo.toml b/crates/openshell-cli/Cargo.toml index b3a006fdd5..8b86544b70 100644 --- a/crates/openshell-cli/Cargo.toml +++ b/crates/openshell-cli/Cargo.toml @@ -44,10 +44,11 @@ bytes = { workspace = true } http-body-util = { workspace = true } hyper = { workspace = true } hyper-util = { workspace = true } -hyper-rustls = { version = "0.27", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "ring", "webpki-tokio"] } +hyper-rustls = { version = "0.27", default-features = false, features = ["native-tokio", "http1", "http2", "tls12", "logging", "ring", "webpki-tokio"] } rustls = { workspace = true } rustls-pemfile = { workspace = true } tokio-rustls = { workspace = true } +tower = { workspace = true } reqwest = { workspace = true } # Error handling @@ -58,6 +59,10 @@ anyhow = { workspace = true } # File archiving (tar-over-SSH sync) tar = "0.4" +# OIDC/Auth +oauth2 = "5" +base64 = { workspace = true } + # WebSocket (Cloudflare tunnel proxy) tokio-tungstenite = { workspace = true } diff --git a/crates/openshell-cli/src/auth.rs b/crates/openshell-cli/src/auth.rs index 6325ebf9c4..8eb421535c 100644 --- a/crates/openshell-cli/src/auth.rs +++ b/crates/openshell-cli/src/auth.rs @@ -72,6 +72,9 @@ fn generate_confirmation_code() -> String { let hash_b = hasher_b.finish(); prev_hash = hash_b; + // hash_b is `u64`; truncation to `usize` is acceptable here since we mod + // by charset.len() (small) and only use it as an index. + #[allow(clippy::cast_possible_truncation)] let idx = (hash_b as usize) % charset.len(); code.push(charset[idx] as char); } @@ -86,6 +89,19 @@ fn generate_confirmation_code() -> String { /// 4. Waits for the XHR POST callback with the CF JWT and matching code /// 5. Returns the token pub async fn browser_auth_flow(gateway_endpoint: &str) -> Result { + // Short-circuit when the browser is suppressed (CI, e2e tests, headless + // environments). Without this early return the function still binds a TCP + // listener, spawns a callback server, and waits the full AUTH_TIMEOUT + // (120 s) for a POST that will never arrive. + let no_browser = std::env::var("OPENSHELL_NO_BROWSER") + .is_ok_and(|v| v == "1" || v.eq_ignore_ascii_case("true")); + if no_browser { + return Err(miette::miette!( + "authentication skipped (OPENSHELL_NO_BROWSER is set).\n\ + Authenticate later with: openshell gateway login" + )); + } + let listener = TcpListener::bind("127.0.0.1:0").await.into_diagnostic()?; let local_addr = listener.local_addr().into_diagnostic()?; let callback_port = local_addr.port(); @@ -108,37 +124,25 @@ pub async fn browser_auth_flow(gateway_endpoint: &str) -> Result { gateway_endpoint.to_string(), )); - // Allow suppressing the browser popup via environment variable (useful for - // CI, e2e tests, and headless environments). - let no_browser = std::env::var("OPENSHELL_NO_BROWSER") - .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) - .unwrap_or(false); - // Prompt the user before opening the browser. eprintln!(" Confirmation code: {code}"); eprintln!(" Verify this code matches your browser before clicking Connect."); eprintln!(); - if no_browser { - eprintln!("Browser opening suppressed (OPENSHELL_NO_BROWSER is set)."); + eprint!("Press Enter to open the browser for authentication..."); + std::io::stderr().flush().ok(); + let mut input = String::new(); + std::io::stdin().read_line(&mut input).ok(); + drop(input); + + if let Err(e) = open_browser_url(&auth_url) { + debug!(error = %e, "failed to open browser"); + eprintln!("Could not open browser automatically."); eprintln!("Open this URL in your browser:"); eprintln!(" {auth_url}"); eprintln!(); } else { - eprint!("Press Enter to open the browser for authentication..."); - std::io::stderr().flush().ok(); - let mut _input = String::new(); - std::io::stdin().read_line(&mut _input).ok(); - - if let Err(e) = open_browser(&auth_url) { - debug!(error = %e, "failed to open browser"); - eprintln!("Could not open browser automatically."); - eprintln!("Open this URL in your browser:"); - eprintln!(" {auth_url}"); - eprintln!(); - } else { - eprintln!("Browser opened."); - } + eprintln!("Browser opened."); } // Wait for the callback or timeout. @@ -163,7 +167,7 @@ pub async fn browser_auth_flow(gateway_endpoint: &str) -> Result { } /// Open a URL in the default browser. -fn open_browser(url: &str) -> std::result::Result<(), String> { +pub fn open_browser_url(url: &str) -> std::result::Result<(), String> { #[cfg(target_os = "macos")] { std::process::Command::new("open") diff --git a/crates/openshell-cli/src/bootstrap.rs b/crates/openshell-cli/src/bootstrap.rs index ee9a481aa2..fcb7744ab4 100644 --- a/crates/openshell-cli/src/bootstrap.rs +++ b/crates/openshell-cli/src/bootstrap.rs @@ -101,7 +101,7 @@ fn is_connectivity_error(error: &miette::Report) -> bool { /// `false` to skip bootstrap. Otherwise returns `true` — a gateway is created /// automatically without prompting the user. pub fn confirm_bootstrap(override_value: Option) -> Result { - if let Some(false) = override_value { + if override_value == Some(false) { return Ok(false); } Ok(true) @@ -191,13 +191,22 @@ pub async fn run_bootstrap( // Deploy the gateway. The deploy flow auto-resumes from existing state // when it finds one. If that fails, fall back to a full recreate. - let handle = match deploy_gateway_with_panel(build_options(false), &gateway_name, location) - .await + let handle = match Box::pin(deploy_gateway_with_panel( + build_options(false), + &gateway_name, + location, + )) + .await { Ok(handle) => handle, Err(resume_err) => { tracing::warn!("auto-bootstrap resume failed, falling back to recreate: {resume_err}"); - deploy_gateway_with_panel(build_options(true), &gateway_name, location).await? + Box::pin(deploy_gateway_with_panel( + build_options(true), + &gateway_name, + location, + )) + .await? } }; let server = handle.gateway_endpoint().to_string(); diff --git a/crates/openshell-cli/src/completers.rs b/crates/openshell-cli/src/completers.rs index 3c2a8b3360..d8ba3ff938 100644 --- a/crates/openshell-cli/src/completers.rs +++ b/crates/openshell-cli/src/completers.rs @@ -7,6 +7,7 @@ use std::time::Duration; use clap_complete::engine::CompletionCandidate; use openshell_bootstrap::{list_gateways, load_active_gateway, load_gateway_metadata}; +use openshell_core::ObjectName; use openshell_core::proto::open_shell_client::OpenShellClient; use openshell_core::proto::{ListProvidersRequest, ListSandboxesRequest}; use tonic::transport::{Channel, Endpoint}; @@ -33,6 +34,7 @@ pub fn complete_sandbox_names(_prefix: &OsStr) -> Vec { .list_sandboxes(ListSandboxesRequest { limit: 200, offset: 0, + label_selector: String::new(), }) .await .ok()?; @@ -41,7 +43,7 @@ pub fn complete_sandbox_names(_prefix: &OsStr) -> Vec { .into_inner() .sandboxes .into_iter() - .map(|s| CompletionCandidate::new(s.name)) + .map(|s| CompletionCandidate::new(s.object_name())) .collect(), ) }) @@ -64,7 +66,7 @@ pub fn complete_provider_names(_prefix: &OsStr) -> Vec { .into_inner() .providers .into_iter() - .map(|p| CompletionCandidate::new(p.name)) + .map(|p| CompletionCandidate::new(p.object_name())) .collect(), ) }) @@ -175,22 +177,19 @@ mod tests { name: "alpha".to_string(), gateway_endpoint: "https://alpha.example.com".to_string(), is_remote: true, - gateway_port: 0, - remote_host: None, - resolved_host: None, auth_mode: Some("cloudflare_jwt".to_string()), - edge_team_domain: None, - edge_auth_url: None, + client_lifecycle_managed: Some(false), + ..Default::default() }, ) .unwrap(); let result = complete_gateway_names(OsStr::new("a")); - let names: Vec = result - .iter() - .map(|candidate| candidate.get_value().to_string_lossy().into_owned()) - .collect(); - assert!(names.contains(&"alpha".to_string())); + assert!( + result + .iter() + .any(|candidate| candidate.get_value().to_string_lossy() == "alpha") + ); }); } } diff --git a/crates/openshell-cli/src/doctor_llm_prompt.md b/crates/openshell-cli/src/doctor_llm_prompt.md index 4d4a6b64c7..fbb26a5656 100644 --- a/crates/openshell-cli/src/doctor_llm_prompt.md +++ b/crates/openshell-cli/src/doctor_llm_prompt.md @@ -68,11 +68,13 @@ Before running commands, establish: ### Step 0: Quick Connectivity Check Run `openshell status` first. This immediately reveals: + - Which gateway and endpoint the CLI is targeting - Whether the CLI can reach the server (mTLS handshake success/failure) - The server version if connected Common errors at this stage: + - **`tls handshake eof`**: The server isn't running or mTLS credentials are missing/mismatched - **`connection refused`**: The container isn't running or port mapping is broken - **`No gateway configured`**: No gateway has been deployed yet @@ -222,7 +224,8 @@ openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o ``` Common mTLS issues: -- **Secrets missing**: The `openshell` namespace may not have been created yet (Helm controller race). Bootstrap waits up to 2 minutes for the namespace. + +- **Secrets missing**: The `openshell` namespace may not have been created yet (Helm controller race). Bootstrap waits up to about 5 minutes for the namespace. - **mTLS mismatch after manual secret deletion**: Delete all three secrets and redeploy — bootstrap will regenerate and restart the workload. - **CLI can't connect after redeploy**: Check that `~/.config/openshell/gateways//mtls/` contains `ca.crt`, `tls.crt`, `tls.key` and that they were updated at deploy time. - **Local mTLS files missing**: The gateway was deployed but CLI credentials weren't persisted (e.g., interrupted deploy). Extract from the cluster secret as shown above. diff --git a/crates/openshell-cli/src/lib.rs b/crates/openshell-cli/src/lib.rs index 09e05449b9..d518557b7e 100644 --- a/crates/openshell-cli/src/lib.rs +++ b/crates/openshell-cli/src/lib.rs @@ -12,6 +12,8 @@ pub mod auth; pub mod bootstrap; pub mod completers; pub mod edge_tunnel; +pub mod oidc_auth; +pub(crate) mod policy_update; pub mod run; pub mod ssh; pub mod tls; diff --git a/crates/openshell-cli/src/main.rs b/crates/openshell-cli/src/main.rs index 2929224118..295ce98a22 100644 --- a/crates/openshell-cli/src/main.rs +++ b/crates/openshell-cli/src/main.rs @@ -122,17 +122,52 @@ fn resolve_gateway_name(gateway_flag: &Option) -> Option { .or_else(load_active_gateway) } -/// Apply edge authentication token from local storage when the gateway uses edge auth. +/// Apply authentication token from local storage based on gateway auth mode. /// -/// When the resolved gateway has `auth_mode == "cloudflare_jwt"`, loads the -/// stored edge token from disk and sets it on the `TlsOptions`. The token is -/// always read from gateway metadata rather than supplied via a CLI flag. -fn apply_edge_auth(tls: &mut TlsOptions, gateway_name: &str) { - if let Some(meta) = get_gateway_metadata(gateway_name) - && meta.auth_mode.as_deref() == Some("cloudflare_jwt") - && let Some(token) = load_edge_token(gateway_name) - { - tls.edge_token = Some(token); +/// Handles both Cloudflare Access (`edge_token`) and OIDC (`oidc_token`) +/// auth modes by loading the stored token and setting it on `TlsOptions`. +/// For OIDC, automatically refreshes the token if it's near expiry. +fn apply_auth(tls: &mut TlsOptions, gateway_name: &str) { + let Some(meta) = get_gateway_metadata(gateway_name) else { + return; + }; + match meta.auth_mode.as_deref() { + Some("cloudflare_jwt") => { + if let Some(token) = load_edge_token(gateway_name) { + tls.edge_token = Some(token); + } + } + Some("oidc") => { + let Some(bundle) = openshell_bootstrap::oidc_token::load_oidc_token(gateway_name) + else { + return; + }; + if openshell_bootstrap::oidc_token::is_token_expired(&bundle) { + // Try to refresh the token in-place using block_in_place + // so the async refresh can run within the sync apply_auth call. + match tokio::task::block_in_place(|| { + tokio::runtime::Handle::current() + .block_on(openshell_cli::oidc_auth::oidc_refresh_token(&bundle)) + }) { + Ok(refreshed) => { + let _ = openshell_bootstrap::oidc_token::store_oidc_token( + gateway_name, + &refreshed, + ); + tls.oidc_token = Some(refreshed.access_token); + } + Err(e) => { + tracing::warn!("OIDC token refresh failed: {e}"); + // Use the expired token anyway — server will reject it + // with a clear error prompting re-login. + tls.oidc_token = Some(bundle.access_token); + } + } + } else { + tls.oidc_token = Some(bundle.access_token); + } + } + _ => {} } } @@ -151,7 +186,7 @@ fn resolve_sandbox_name(name: Option, gateway: &str) -> Result { Specify a sandbox name or connect to one first: nav sandbox connect " ) })?; - eprintln!("{} Using sandbox '{}' (last used)", "→".bold(), last.bold(),); + eprintln!("{} Using sandbox '{}' (last used)", "→".bold(), last.bold()); Ok(last) } @@ -254,6 +289,8 @@ const POLICY_EXAMPLES: &str = "\x1b[1mALIAS\x1b[0m \x1b[1mEXAMPLES\x1b[0m $ openshell policy get my-sandbox $ openshell policy set my-sandbox --policy policy.yaml + $ openshell policy update my-sandbox --add-endpoint api.github.com:443:read-only:rest:enforce + $ openshell policy update my-sandbox --add-allow 'api.github.com:443:GET:/repos/**' $ openshell policy set --global --policy policy.yaml $ openshell policy delete --global $ openshell policy list my-sandbox @@ -262,11 +299,10 @@ const POLICY_EXAMPLES: &str = "\x1b[1mALIAS\x1b[0m const SETTINGS_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m $ openshell settings get my-sandbox $ openshell settings get --global - $ openshell settings set my-sandbox --key log_level --value debug - $ openshell settings set --global --key log_level --value warn - $ openshell settings set --global --key dummy_bool --value yes - $ openshell settings set --global --key dummy_int --value 42 - $ openshell settings delete --global --key log_level + $ openshell settings set --global --key providers_v2_enabled --value true + $ openshell settings set my-sandbox --key ocsf_json_enabled --value true + $ openshell settings set --global --key ocsf_json_enabled --value true + $ openshell settings delete --global --key providers_v2_enabled "; const PROVIDER_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m @@ -343,6 +379,15 @@ struct Cli { )] gateway_endpoint: Option, + /// Skip TLS certificate verification for gateway connections. + #[arg( + long, + global = true, + env = "OPENSHELL_GATEWAY_INSECURE", + help_heading = "GATEWAY FLAGS" + )] + gateway_insecure: bool, + /// Increase verbosity (-v, -vv, -vvv). #[arg(short, long, action = clap::ArgAction::Count, global = true, help_heading = "GLOBAL FLAGS")] verbose: u8, @@ -698,6 +743,10 @@ enum ProviderCommands { names: bool, }, + /// List available provider profiles. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + ListProfiles, + /// Update an existing provider's credentials or config. #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] Update { @@ -814,11 +863,45 @@ enum GatewayCommands { /// `nvidia.com/gpu` resources. Requires NVIDIA drivers and the /// NVIDIA Container Toolkit on the host. /// - /// When enabled, OpenShell auto-selects CDI when the Docker daemon has + /// When enabled, `OpenShell` auto-selects CDI when the Docker daemon has /// CDI enabled and falls back to Docker's NVIDIA GPU request path /// (`--gpus all`) otherwise. #[arg(long)] gpu: bool, + + /// OIDC issuer URL for JWT-based authentication. + /// When set, the K3s server will validate Bearer tokens against this issuer. + #[arg(long)] + oidc_issuer: Option, + + /// OIDC audience for the API resource server. + #[arg(long, default_value = "openshell-cli", requires = "oidc_issuer")] + oidc_audience: String, + + /// OIDC client ID stored in gateway metadata for CLI login. + #[arg(long, default_value = "openshell-cli", requires = "oidc_issuer")] + oidc_client_id: String, + + /// Dot-separated path to the roles array in the JWT claims. + #[arg(long, requires = "oidc_issuer")] + oidc_roles_claim: Option, + + /// Role name that grants admin access. + #[arg(long, requires = "oidc_issuer")] + oidc_admin_role: Option, + + /// Role name that grants standard user access. + #[arg(long, requires = "oidc_issuer")] + oidc_user_role: Option, + + /// Space-separated `OAuth2` scopes to request during OIDC login. + #[arg(long, requires = "oidc_issuer")] + oidc_scopes: Option, + + /// Dot-separated path to the scopes value in the JWT claims. + /// When set, the server enforces scope-based permissions on top of roles. + #[arg(long, requires = "oidc_issuer")] + oidc_scopes_claim: Option, }, /// Stop the gateway (preserves state). @@ -895,9 +978,29 @@ enum GatewayCommands { /// With `http://...`, stores a local plaintext registration instead. #[arg(long, conflicts_with = "remote")] local: bool, + + /// Register as an OIDC-authenticated gateway using the given issuer URL. + /// The server must be configured with `--oidc-issuer` matching this URL. + #[arg(long, conflicts_with = "remote")] + oidc_issuer: Option, + + /// OIDC client ID for the CLI login flow (defaults to "openshell-cli"). + #[arg(long, default_value = "openshell-cli", requires = "oidc_issuer")] + oidc_client_id: String, + + /// OIDC audience for the API resource server. When different from + /// the client ID, the CLI requests this audience in the token exchange. + /// Defaults to the client ID value. + #[arg(long, requires = "oidc_issuer")] + oidc_audience: Option, + + /// Space-separated `OAuth2` scopes to request during OIDC login. + /// When set, tokens will include these scopes for fine-grained access control. + #[arg(long, requires = "oidc_issuer")] + oidc_scopes: Option, }, - /// Authenticate with an edge-authenticated gateway. + /// Authenticate with an edge-authenticated or OIDC gateway. /// /// Opens a browser for the edge proxy's login flow and stores the /// token locally. Use this to re-authenticate when a token expires. @@ -908,6 +1011,17 @@ enum GatewayCommands { name: Option, }, + /// Clear stored authentication credentials for a gateway. + /// + /// Removes the locally stored OIDC token or edge token so subsequent + /// commands require re-authentication via `gateway login`. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + Logout { + /// Gateway name (defaults to the active gateway). + #[arg(add = ArgValueCompleter::new(completers::complete_gateway_names))] + name: Option, + }, + /// Select the active gateway. /// /// When called without a name, opens an interactive chooser on a TTY and @@ -926,6 +1040,13 @@ enum GatewayCommands { #[arg(long, env = "OPENSHELL_GATEWAY", add = ArgValueCompleter::new(completers::complete_gateway_names))] name: Option, }, + + /// List registered gateways. + /// + /// Prints a table of all registered gateways with their endpoint, type, + /// and authentication mode. The active gateway is marked with `*`. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + List, } // ----------------------------------------------------------------------- @@ -1083,7 +1204,7 @@ enum SandboxCommands { #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] Create { /// Optional sandbox name (auto-generated when omitted). - #[arg(long)] + #[arg(long, add = ArgValueCompleter::new(completers::complete_sandbox_names))] name: Option, /// Sandbox source: a community sandbox name (e.g., `openclaw`), a path @@ -1136,6 +1257,11 @@ enum SandboxCommands { #[arg(long)] gpu: bool, + /// Target a specific GPU by PCI address (e.g. "0000:2d:00.0") or index (e.g. "0", "1"). + /// Only valid with --gpu. When omitted with --gpu, the first available GPU is assigned. + #[arg(long, requires = "gpu")] + gpu_device: Option, + /// SSH destination for remote bootstrap (e.g., user@hostname). /// Only used when no cluster exists yet; ignored if a cluster is /// already active. @@ -1156,7 +1282,7 @@ enum SandboxCommands { policy: Option, /// Forward a local port to the sandbox before the initial command or shell starts. - /// Accepts [bind_address:]port (e.g. 8080, 0.0.0.0:8080). Keeps the sandbox alive. + /// Accepts [`bind_address`:]port (e.g. 8080, 0.0.0.0:8080). Keeps the sandbox alive. #[arg(long, conflicts_with = "no_keep")] forward: Option, @@ -1195,8 +1321,12 @@ enum SandboxCommands { #[arg(long, overrides_with = "auto_providers")] no_auto_providers: bool, + /// Attach labels to the sandbox (key=value format, repeatable). + #[arg(long = "label")] + labels: Vec, + /// Command to run after "--" (defaults to an interactive shell). - #[arg(trailing_var_arg = true)] + #[arg(last = true, allow_hyphen_values = true)] command: Vec, }, @@ -1206,6 +1336,10 @@ enum SandboxCommands { /// Sandbox name (defaults to last-used sandbox). #[arg(add = ArgValueCompleter::new(completers::complete_sandbox_names))] name: Option, + + /// Print only the active policy YAML (same policy as the default view; stdout only). + #[arg(long)] + policy_only: bool, }, /// List sandboxes. @@ -1226,6 +1360,10 @@ enum SandboxCommands { /// Print only sandbox names (one per line). #[arg(long, conflicts_with = "ids")] names: bool, + + /// Filter sandboxes by label selector (key1=value1,key2=value2). + #[arg(long)] + selector: Option, }, /// Delete a sandbox by name. @@ -1438,6 +1576,54 @@ enum PolicyCommands { timeout: u64, }, + /// Incrementally update policy on a live sandbox. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + Update { + /// Sandbox name (defaults to last-used sandbox). + #[arg(add = ArgValueCompleter::new(completers::complete_sandbox_names))] + name: Option, + + /// Add or merge an endpoint: host:port[:access[:protocol[:enforcement]]]. + #[arg(long = "add-endpoint")] + add_endpoints: Vec, + + /// Remove an endpoint: host:port. + #[arg(long = "remove-endpoint")] + remove_endpoints: Vec, + + /// Add a REST allow rule: `host:port:METHOD:path_glob`. + #[arg(long = "add-allow")] + add_allow: Vec, + + /// Add a REST deny rule: `host:port:METHOD:path_glob`. + #[arg(long = "add-deny")] + add_deny: Vec, + + /// Remove a network rule by name. + #[arg(long = "remove-rule")] + remove_rules: Vec, + + /// Add binaries to each --add-endpoint rule. + #[arg(long = "binary", value_hint = ValueHint::FilePath)] + binaries: Vec, + + /// Override the generated rule name when exactly one --add-endpoint is provided. + #[arg(long = "rule-name")] + rule_name: Option, + + /// Preview the merged policy without sending it to the gateway. + #[arg(long)] + dry_run: bool, + + /// Wait for the sandbox to load the policy revision. + #[arg(long)] + wait: bool, + + /// Timeout for --wait in seconds. + #[arg(long, default_value_t = 60)] + timeout: u64, + }, + /// Show current active policy for a sandbox or the global policy. #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] Get { @@ -1489,7 +1675,7 @@ enum PolicyCommands { /// Prove properties of a sandbox policy — or find counterexamples. #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] Prove { - /// Path to OpenShell sandbox policy YAML. + /// Path to `OpenShell` sandbox policy YAML. #[arg(long, value_hint = ValueHint::FilePath)] policy: String, @@ -1579,7 +1765,7 @@ enum ForwardCommands { /// Start forwarding a local port to a sandbox. #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] Start { - /// Port to forward: [bind_address:]port (e.g. 8080, 0.0.0.0:8080). + /// Port to forward: [`bind_address`:]port (e.g. 8080, 0.0.0.0:8080). port: String, /// Sandbox name (defaults to last-used sandbox). @@ -1608,6 +1794,7 @@ enum ForwardCommands { } #[tokio::main] +#[allow(clippy::large_stack_frames)] // CLI dispatch holds many futures; OK at top level. async fn main() -> Result<()> { // Install the rustls crypto provider before completion runs — completers may // establish TLS connections to the gateway. @@ -1618,7 +1805,8 @@ async fn main() -> Result<()> { CompleteEnv::with_factory(Cli::command).complete(); let cli = Cli::parse(); - let tls = TlsOptions::default(); + let mut tls = TlsOptions::default(); + tls.gateway_insecure = cli.gateway_insecure; // Set up logging based on verbosity let log_level = match cli.verbose { @@ -1635,6 +1823,23 @@ async fn main() -> Result<()> { ) .init(); + // Propagate verbosity to the OpenSSH LogLevel used by SSH subprocesses. + // Only set the env var when it hasn't been explicitly overridden by the + // user, so `OPENSHELL_SSH_LOG_LEVEL=DEBUG openshell ...` still wins. + if std::env::var("OPENSHELL_SSH_LOG_LEVEL").is_err() { + let ssh_log_level = match cli.verbose { + 0 => "ERROR", + 1 => "INFO", + _ => "DEBUG", + }; + // SAFETY: Called early in main() before spawning async tasks that + // read the environment, so no concurrent readers exist. + #[allow(unsafe_code)] + unsafe { + std::env::set_var("OPENSHELL_SSH_LOG_LEVEL", ssh_log_level); + } + } + match cli.command { // ----------------------------------------------------------- // Gateway commands (was `cluster` / `cluster admin`) @@ -1654,13 +1859,21 @@ async fn main() -> Result<()> { registry_username, registry_token, gpu, + oidc_issuer, + oidc_audience, + oidc_client_id, + oidc_roles_claim, + oidc_admin_role, + oidc_user_role, + oidc_scopes, + oidc_scopes_claim, } => { let gpu = if gpu { vec!["auto".to_string()] } else { vec![] }; - run::gateway_admin_deploy( + Box::pin(run::gateway_admin_deploy( &name, remote.as_deref(), ssh_key.as_deref(), @@ -1672,7 +1885,15 @@ async fn main() -> Result<()> { registry_username.as_deref(), registry_token.as_deref(), gpu, - ) + oidc_issuer.as_deref(), + &oidc_audience, + &oidc_client_id, + oidc_roles_claim.as_deref(), + oidc_admin_role.as_deref(), + oidc_user_role.as_deref(), + oidc_scopes.as_deref(), + oidc_scopes_claim.as_deref(), + )) .await?; } GatewayCommands::Stop { @@ -1701,6 +1922,10 @@ async fn main() -> Result<()> { remote, ssh_key, local, + oidc_issuer, + oidc_client_id, + oidc_audience, + oidc_scopes, } => { run::gateway_add( &endpoint, @@ -1708,6 +1933,10 @@ async fn main() -> Result<()> { remote.as_deref(), ssh_key.as_deref(), local, + oidc_issuer.as_deref(), + &oidc_client_id, + oidc_audience.as_deref(), + oidc_scopes.as_deref(), ) .await?; } @@ -1723,6 +1952,18 @@ async fn main() -> Result<()> { })?; run::gateway_login(&name).await?; } + GatewayCommands::Logout { name } => { + let name = name + .or_else(|| resolve_gateway_name(&cli.gateway)) + .ok_or_else(|| { + miette::miette!( + "No active gateway.\n\ + Specify a gateway name: openshell gateway logout \n\ + Or set one with: openshell gateway select " + ) + })?; + run::gateway_logout(&name)?; + } GatewayCommands::Select { name } => { run::gateway_select(name.as_deref(), &cli.gateway)?; } @@ -1732,6 +1973,9 @@ async fn main() -> Result<()> { .unwrap_or_else(|| "openshell".to_string()); run::gateway_admin_info(&name)?; } + GatewayCommands::List => { + run::gateway_list(&cli.gateway)?; + } }, // ----------------------------------------------------------- @@ -1784,12 +2028,12 @@ async fn main() -> Result<()> { Some(Commands::Status) => { if let Ok(ctx) = resolve_gateway(&cli.gateway, &cli.gateway_endpoint) { let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); run::gateway_status(&ctx.name, &ctx.endpoint, &tls).await?; } else { println!("{}", "Gateway Status".cyan().bold()); println!(); - println!(" {} No gateway configured.", "Status:".dimmed(),); + println!(" {} No gateway configured.", "Status:".dimmed()); println!(); println!( "Deploy a gateway with: {}", @@ -1807,16 +2051,15 @@ async fn main() -> Result<()> { ForwardCommands::Stop { port, name } => { let name = match name { Some(n) => n, - None => match run::find_forward_by_port(port)? { - Some(n) => { + None => { + if let Some(n) = run::find_forward_by_port(port)? { eprintln!("→ Found forward on sandbox '{n}'"); n - } - None => { - eprintln!("{} No active forward found for port {port}", "!".yellow(),); + } else { + eprintln!("{} No active forward found for port {port}", "!".yellow()); return Ok(()); } - }, + } }; if run::stop_forward(&name, port)? { eprintln!( @@ -1883,7 +2126,7 @@ async fn main() -> Result<()> { let spec = openshell_core::forward::ForwardSpec::parse(&port)?; let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); let name = resolve_sandbox_name(name, &ctx.name)?; run::sandbox_forward(&ctx.endpoint, &name, &spec, background, &tls).await?; if background { @@ -1911,7 +2154,7 @@ async fn main() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); let name = resolve_sandbox_name(name, &ctx.name)?; run::sandbox_logs( &ctx.endpoint, @@ -1956,7 +2199,7 @@ async fn main() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); match policy_cmd { PolicyCommands::Set { name, @@ -1988,6 +2231,37 @@ async fn main() -> Result<()> { .await?; } } + PolicyCommands::Update { + name, + add_endpoints, + remove_endpoints, + add_allow, + add_deny, + remove_rules, + binaries, + rule_name, + dry_run, + wait, + timeout, + } => { + let name = resolve_sandbox_name(name, &ctx.name)?; + run::sandbox_policy_update( + &ctx.endpoint, + &name, + &add_endpoints, + &remove_endpoints, + &add_deny, + &add_allow, + &remove_rules, + &binaries, + rule_name.as_deref(), + dry_run, + wait, + timeout, + &tls, + ) + .await?; + } PolicyCommands::Get { name, rev, @@ -2033,7 +2307,7 @@ async fn main() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); match settings_cmd { SettingsCommands::Get { name, global, json } => { @@ -2087,7 +2361,7 @@ async fn main() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); match draft_cmd { DraftCommands::Get { name, status } => { let name = resolve_sandbox_name(name, &ctx.name)?; @@ -2140,7 +2414,7 @@ async fn main() -> Result<()> { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let endpoint = &ctx.endpoint; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); match command { InferenceCommands::Set { provider, @@ -2197,6 +2471,7 @@ async fn main() -> Result<()> { no_keep, editor, gpu, + gpu_device, remote, ssh_key, providers, @@ -2208,6 +2483,7 @@ async fn main() -> Result<()> { no_bootstrap, auto_providers, no_auto_providers, + labels, command, } => { // Resolve --tty / --no-tty into an Option override. @@ -2238,6 +2514,19 @@ async fn main() -> Result<()> { None // prompt or auto-detect }; + // Parse --label flags into a HashMap. + let mut labels_map = std::collections::HashMap::new(); + for label_str in &labels { + let parts: Vec<&str> = label_str.splitn(2, '=').collect(); + if parts.len() != 2 { + return Err(miette::miette!( + "invalid label format '{}', expected key=value", + label_str + )); + } + labels_map.insert(parts[0].to_string(), parts[1].to_string()); + } + // Parse --upload spec into (local_path, sandbox_path, git_ignore). let upload_spec = upload.as_deref().map(|s| { let (local, remote) = parse_upload_spec(s); @@ -2265,7 +2554,7 @@ async fn main() -> Result<()> { } let endpoint = &ctx.endpoint; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); // The user already has a configured gateway. Disable // auto-bootstrap in the retry path so we don't // silently replace their selected gateway with a new @@ -2278,6 +2567,7 @@ async fn main() -> Result<()> { upload_spec.as_ref(), keep, gpu, + gpu_device.as_deref(), editor, remote.as_deref(), ssh_key.as_deref(), @@ -2288,6 +2578,7 @@ async fn main() -> Result<()> { tty_override, Some(false), auto_providers_override, + &labels_map, &tls, )) .await?; @@ -2300,6 +2591,7 @@ async fn main() -> Result<()> { upload_spec.as_ref(), keep, gpu, + gpu_device.as_deref(), editor, remote.as_deref(), ssh_key.as_deref(), @@ -2323,7 +2615,7 @@ async fn main() -> Result<()> { } => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); let sandbox_dest = dest.as_deref(); let local = std::path::Path::new(&local_path); if !local.exists() { @@ -2340,6 +2632,7 @@ async fn main() -> Result<()> { &name, &base_dir, &files, + local, sandbox_dest, &tls, ) @@ -2358,7 +2651,7 @@ async fn main() -> Result<()> { } => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); let local_dest = std::path::Path::new(dest.as_deref().unwrap_or(".")); eprintln!( "Downloading sandbox:{} -> {}", @@ -2373,24 +2666,34 @@ async fn main() -> Result<()> { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let endpoint = &ctx.endpoint; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); match other { SandboxCommands::Create { .. } | SandboxCommands::Upload { .. } | SandboxCommands::Download { .. } => { unreachable!() } - SandboxCommands::Get { name } => { + SandboxCommands::Get { name, policy_only } => { let name = resolve_sandbox_name(name, &ctx.name)?; - run::sandbox_get(endpoint, &name, &tls).await?; + run::sandbox_get(endpoint, &name, policy_only, &tls).await?; } SandboxCommands::List { limit, offset, ids, names, + selector, } => { - run::sandbox_list(endpoint, limit, offset, ids, names, &tls).await?; + run::sandbox_list( + endpoint, + limit, + offset, + ids, + names, + selector.as_deref(), + &tls, + ) + .await?; } SandboxCommands::Delete { names, all } => { run::sandbox_delete(endpoint, &names, all, &tls, &ctx.name).await?; @@ -2453,7 +2756,7 @@ async fn main() -> Result<()> { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let endpoint = &ctx.endpoint; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); match command { ProviderCommands::Create { @@ -2484,6 +2787,9 @@ async fn main() -> Result<()> { } => { run::provider_list(endpoint, limit, offset, names, &tls).await?; } + ProviderCommands::ListProfiles => { + run::provider_list_profiles(endpoint, &tls).await?; + } ProviderCommands::Update { name, from_existing, @@ -2508,7 +2814,7 @@ async fn main() -> Result<()> { Some(Commands::Term { theme }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_edge_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name); let channel = openshell_cli::tls::build_channel(&ctx.endpoint, &tls).await?; openshell_tui::run(channel, &ctx.name, &ctx.endpoint, theme).await?; } @@ -2540,7 +2846,7 @@ async fn main() -> Result<()> { None => tls, }; if let Some(ref g) = gateway_name_opt { - apply_edge_auth(&mut effective_tls, g); + apply_auth(&mut effective_tls, g); } run::sandbox_ssh_proxy(&gw, &sid, &tok, &effective_tls).await?; } @@ -2559,7 +2865,7 @@ async fn main() -> Result<()> { meta.gateway_endpoint }; let mut tls = tls.with_gateway_name(&g); - apply_edge_auth(&mut tls, &g); + apply_auth(&mut tls, &g); run::sandbox_ssh_proxy_by_name(&endpoint, &n, &tls).await?; } // Legacy name mode with --server only (no --gateway-name). @@ -2695,12 +3001,9 @@ mod tests { name: name.to_string(), gateway_endpoint: endpoint.to_string(), is_remote: true, - gateway_port: 0, - remote_host: None, - resolved_host: None, auth_mode: Some("cloudflare_jwt".to_string()), - edge_team_domain: None, - edge_auth_url: None, + client_lifecycle_managed: Some(false), + ..Default::default() } } @@ -3119,7 +3422,7 @@ mod tests { } #[test] - fn apply_edge_auth_uses_stored_token() { + fn apply_auth_uses_stored_token() { let tmp = tempfile::tempdir().unwrap(); with_tmp_xdg(tmp.path(), || { store_gateway_metadata( @@ -3130,13 +3433,13 @@ mod tests { store_edge_token("edge-gateway", "token-123").unwrap(); let mut tls = TlsOptions::default(); - apply_edge_auth(&mut tls, "edge-gateway"); + apply_auth(&mut tls, "edge-gateway"); assert_eq!(tls.edge_token.as_deref(), Some("token-123")); }); } - /// Verify the flag names the TUI uses to build its ProxyCommand are + /// Verify the flag names the TUI uses to build its `ProxyCommand` are /// accepted by the `SshProxy` subcommand and land in the right fields. /// This catches drift when CLI flags are renamed or restructured. #[test] @@ -3178,6 +3481,19 @@ mod tests { } } + #[test] + fn provider_list_profiles_parses() { + let cli = Cli::try_parse_from(["openshell", "provider", "list-profiles"]) + .expect("provider list-profiles should parse"); + + assert!(matches!( + cli.command, + Some(Commands::Provider { + command: Some(ProviderCommands::ListProfiles) + }) + )); + } + #[test] fn settings_set_global_parses_yes_flag() { let cli = Cli::try_parse_from([ @@ -3273,6 +3589,58 @@ mod tests { } } + // ── sandbox create arg-shape tests ─────────────────────────────────── + + /// Verify that `sandbox create --name ` still parses as a named + /// option rather than a positional argument. The `ArgValueCompleter` + /// attribute must be combined with `long` on the `name` field; without + /// `long`, clap treats the field as positional and `--name` is rejected. + #[test] + fn sandbox_create_name_is_a_named_flag() { + // Build the CLI app and try to parse `sandbox create --name foo`. + // `try_parse_from` returns Err if clap rejects the arguments. + let result = Cli::try_parse_from(["openshell", "sandbox", "create", "--name", "my-sb"]); + assert!( + result.is_ok(), + "sandbox create --name should parse as a named flag: {:?}", + result.err() + ); + if let Ok(cli) = result { + if let Some(Commands::Sandbox { + command: Some(SandboxCommands::Create { name, .. }), + .. + }) = cli.command + { + assert_eq!(name.as_deref(), Some("my-sb")); + } else { + panic!("expected SandboxCommands::Create"); + } + } + } + + /// Verify that `sandbox create` without `--name` still works (name is + /// optional and auto-generated when omitted). + #[test] + fn sandbox_create_name_is_optional() { + let result = Cli::try_parse_from(["openshell", "sandbox", "create"]); + assert!( + result.is_ok(), + "sandbox create without --name should be accepted: {:?}", + result.err() + ); + if let Ok(cli) = result { + if let Some(Commands::Sandbox { + command: Some(SandboxCommands::Create { name, .. }), + .. + }) = cli.command + { + assert_eq!(name, None); + } else { + panic!("expected SandboxCommands::Create"); + } + } + } + /// Ensure every provider registered in `ProviderRegistry` has a /// corresponding `CliProviderType` variant (and vice-versa). /// This test would have caught the missing `Copilot` variant from #707. diff --git a/crates/openshell-cli/src/oidc_auth.rs b/crates/openshell-cli/src/oidc_auth.rs new file mode 100644 index 0000000000..fd742a418d --- /dev/null +++ b/crates/openshell-cli/src/oidc_auth.rs @@ -0,0 +1,438 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! OIDC authentication flows for CLI gateway login. +//! +//! Implements Authorization Code + PKCE (interactive browser flow) and +//! Client Credentials (CI/automation) `OAuth2` grant types against a +//! Keycloak-compatible OIDC provider. + +use bytes::Bytes; +use http_body_util::Full; +use hyper::service::service_fn; +use hyper::{Method, Response, StatusCode}; +use hyper_util::rt::{TokioExecutor, TokioIo}; +use hyper_util::server::conn::auto::Builder; +use miette::{IntoDiagnostic, Result}; +use oauth2::basic::BasicClient; +use oauth2::{ + AuthType, AuthUrl, AuthorizationCode, ClientId, ClientSecret, CsrfToken, PkceCodeChallenge, + RedirectUrl, RefreshToken, Scope, TokenResponse, TokenUrl, +}; +use openshell_bootstrap::oidc_token::OidcTokenBundle; +use serde::Deserialize; +use std::convert::Infallible; +use std::sync::{Arc, Mutex}; +use std::time::Duration; +use tokio::net::TcpListener; +use tokio::sync::oneshot; +use tracing::debug; + +const AUTH_TIMEOUT: Duration = Duration::from_secs(120); + +/// OIDC discovery document (subset of fields we need). +#[derive(Debug, Deserialize)] +struct OidcDiscovery { + issuer: String, + authorization_endpoint: String, + token_endpoint: String, +} + +/// Discover OIDC endpoints from the issuer's well-known configuration. +/// +/// Validates that the discovery document's `issuer` field matches the +/// configured issuer URL to prevent SSRF or misdirection. +async fn discover(issuer: &str) -> Result { + let normalized_issuer = issuer.trim_end_matches('/'); + let url = format!("{normalized_issuer}/.well-known/openid-configuration"); + let resp: OidcDiscovery = reqwest::get(&url) + .await + .into_diagnostic()? + .json() + .await + .into_diagnostic()?; + + let discovered_issuer = resp.issuer.trim_end_matches('/'); + if discovered_issuer != normalized_issuer { + return Err(miette::miette!( + "OIDC discovery issuer mismatch: expected '{}', got '{}'", + normalized_issuer, + discovered_issuer + )); + } + Ok(resp) +} + +fn http_client() -> reqwest::Client { + reqwest::ClientBuilder::new() + .redirect(reqwest::redirect::Policy::none()) + .build() + .expect("failed to build HTTP client") +} + +fn build_scopes(scopes: Option<&str>) -> Vec { + let mut result = vec![Scope::new("openid".to_string())]; + if let Some(s) = scopes { + for scope in s.split_whitespace() { + if scope != "openid" { + result.push(Scope::new(scope.to_string())); + } + } + } + result +} + +fn build_ci_scopes(scopes: Option<&str>) -> Vec { + let Some(s) = scopes else { + return vec![]; + }; + s.split_whitespace() + .map(|scope| Scope::new(scope.to_string())) + .collect() +} + +/// Run the OIDC Authorization Code + PKCE browser flow. +/// +/// Opens the user's browser to the Keycloak login page and waits for +/// the authorization code redirect on a localhost callback server. +pub async fn oidc_browser_auth_flow( + issuer: &str, + client_id: &str, + audience: Option<&str>, + scopes: Option<&str>, +) -> Result { + let discovery = discover(issuer).await?; + + let listener = TcpListener::bind("127.0.0.1:0").await.into_diagnostic()?; + let port = listener.local_addr().into_diagnostic()?.port(); + let redirect_uri = format!("http://127.0.0.1:{port}/callback"); + + let client = BasicClient::new(ClientId::new(client_id.to_string())) + .set_auth_uri(AuthUrl::new(discovery.authorization_endpoint).into_diagnostic()?) + .set_token_uri(TokenUrl::new(discovery.token_endpoint).into_diagnostic()?) + .set_redirect_uri(RedirectUrl::new(redirect_uri).into_diagnostic()?); + + let (pkce_challenge, pkce_verifier) = PkceCodeChallenge::new_random_sha256(); + + let mut auth_request = client + .authorize_url(CsrfToken::new_random) + .set_pkce_challenge(pkce_challenge); + + for scope in build_scopes(scopes) { + auth_request = auth_request.add_scope(scope); + } + + let (mut auth_url, csrf_token) = auth_request.url(); + + // Append audience parameter for providers like Entra ID where the API + // audience differs from the client ID. + if let Some(aud) = audience { + auth_url.query_pairs_mut().append_pair("audience", aud); + } + + let (tx, rx) = oneshot::channel::(); + let expected_state = csrf_token.secret().clone(); + + let server_handle = tokio::spawn(run_oidc_callback_server(listener, tx, expected_state)); + + eprintln!(" Opening browser for OIDC authentication..."); + if let Err(e) = crate::auth::open_browser_url(auth_url.as_str()) { + debug!(error = %e, "failed to open browser"); + eprintln!("Could not open browser automatically."); + eprintln!("Open this URL in your browser:"); + eprintln!(" {auth_url}"); + eprintln!(); + } else { + eprintln!(" Browser opened. Waiting for authentication..."); + } + + let code = tokio::select! { + result = rx => { + result.map_err(|_| miette::miette!("OIDC callback channel closed unexpectedly"))? + } + () = tokio::time::sleep(AUTH_TIMEOUT) => { + return Err(miette::miette!( + "OIDC authentication timed out after {} seconds.\n\ + Try again with: openshell gateway login", + AUTH_TIMEOUT.as_secs() + )); + } + }; + + server_handle.abort(); + + let http = http_client(); + let token_response = client + .exchange_code(AuthorizationCode::new(code)) + .set_pkce_verifier(pkce_verifier) + .request_async(&http) + .await + .map_err(|e| miette::miette!("token exchange failed: {e}"))?; + + Ok(bundle_from_oauth2_response( + &token_response, + issuer, + client_id, + )) +} + +/// Run the OIDC Client Credentials flow (for CI/automation). +/// +/// Reads `OPENSHELL_OIDC_CLIENT_SECRET` from the environment. +pub async fn oidc_client_credentials_flow( + issuer: &str, + client_id: &str, + audience: Option<&str>, + scopes: Option<&str>, +) -> Result { + let client_secret = std::env::var("OPENSHELL_OIDC_CLIENT_SECRET").map_err(|_| { + miette::miette!( + "OPENSHELL_OIDC_CLIENT_SECRET environment variable is required for client credentials flow" + ) + })?; + + let discovery = discover(issuer).await?; + + let client = BasicClient::new(ClientId::new(client_id.to_string())) + .set_client_secret(ClientSecret::new(client_secret)) + .set_token_uri(TokenUrl::new(discovery.token_endpoint).into_diagnostic()?) + .set_auth_type(AuthType::RequestBody); + + let mut request = client.exchange_client_credentials(); + for scope in build_ci_scopes(scopes) { + request = request.add_scope(scope); + } + if let Some(aud) = audience { + request = request.add_extra_param("audience", aud); + } + + let http = http_client(); + let token_response = request + .request_async(&http) + .await + .map_err(|e| miette::miette!("client credentials token exchange failed: {e}"))?; + + Ok(bundle_from_oauth2_response( + &token_response, + issuer, + client_id, + )) +} + +/// Refresh an OIDC token using the `refresh_token` grant. +/// +/// Preserves the existing refresh token if the server does not return a new +/// one (per OAuth 2.0 spec, the refresh response may omit `refresh_token`). +pub async fn oidc_refresh_token(bundle: &OidcTokenBundle) -> Result { + let refresh_token = bundle.refresh_token.as_deref().ok_or_else(|| { + miette::miette!( + "no refresh token available — re-authenticate with: openshell gateway login" + ) + })?; + + let discovery = discover(&bundle.issuer).await?; + + let client = BasicClient::new(ClientId::new(bundle.client_id.clone())) + .set_token_uri(TokenUrl::new(discovery.token_endpoint).into_diagnostic()?); + + let http = http_client(); + let token_response = client + .exchange_refresh_token(&RefreshToken::new(refresh_token.to_string())) + .request_async(&http) + .await + .map_err(|e| miette::miette!("token refresh failed: {e}"))?; + + let mut refreshed = + bundle_from_oauth2_response(&token_response, &bundle.issuer, &bundle.client_id); + if refreshed.refresh_token.is_none() { + refreshed.refresh_token.clone_from(&bundle.refresh_token); + } + Ok(refreshed) +} + +/// Ensure we have a valid OIDC token for the given gateway, refreshing if needed. +/// +/// Returns the access token string. +pub async fn ensure_valid_oidc_token(gateway_name: &str) -> Result { + let bundle = + openshell_bootstrap::oidc_token::load_oidc_token(gateway_name).ok_or_else(|| { + miette::miette!( + "No OIDC token stored for gateway '{gateway_name}'.\n\ + Authenticate with: openshell gateway login" + ) + })?; + + if !openshell_bootstrap::oidc_token::is_token_expired(&bundle) { + return Ok(bundle.access_token); + } + + debug!( + gateway = gateway_name, + "OIDC token expired, attempting refresh" + ); + let refreshed = oidc_refresh_token(&bundle).await?; + openshell_bootstrap::oidc_token::store_oidc_token(gateway_name, &refreshed)?; + Ok(refreshed.access_token) +} + +// ── Helpers ────────────────────────────────────────────────────────── + +fn bundle_from_oauth2_response( + resp: &oauth2::basic::BasicTokenResponse, + issuer: &str, + client_id: &str, +) -> OidcTokenBundle { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_secs(); + + OidcTokenBundle { + access_token: resp.access_token().secret().clone(), + refresh_token: resp.refresh_token().map(|rt| rt.secret().clone()), + expires_at: resp.expires_in().map(|ei| now + ei.as_secs()), + issuer: issuer.to_string(), + client_id: client_id.to_string(), + } +} + +/// Percent-decode a URL query parameter value. +fn percent_decode(s: &str) -> String { + let mut out = Vec::with_capacity(s.len()); + let mut bytes = s.bytes(); + while let Some(b) = bytes.next() { + if b == b'%' { + let hi = bytes.next().and_then(|b| char::from(b).to_digit(16)); + let lo = bytes.next().and_then(|b| char::from(b).to_digit(16)); + if let (Some(h), Some(l)) = (hi, lo) { + out.push(u8::try_from(h * 16 + l).unwrap_or(b'%')); + } else { + out.push(b'%'); + } + } else if b == b'+' { + out.push(b' '); + } else { + out.push(b); + } + } + String::from_utf8(out).unwrap_or_else(|_| s.to_string()) +} + +/// Callback server state. +struct CallbackState { + expected_state: String, + tx: Mutex>>, +} + +impl CallbackState { + fn take_sender(&self) -> Option> { + self.tx + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .take() + } +} + +/// Run the ephemeral callback server for the OIDC redirect. +/// +/// Listens for `GET /callback?code=...&state=...`. +async fn run_oidc_callback_server( + listener: TcpListener, + tx: oneshot::Sender, + expected_state: String, +) { + let state = Arc::new(CallbackState { + expected_state, + tx: Mutex::new(Some(tx)), + }); + + loop { + let Ok((stream, _)) = listener.accept().await else { + return; + }; + let state = Arc::clone(&state); + tokio::spawn(async move { + let service = service_fn(move |req| { + let state = Arc::clone(&state); + async move { Ok::<_, Infallible>(handle_oidc_callback(req, state)) } + }); + + if let Err(error) = Builder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(stream), service) + .await + { + debug!(error = %error, "OIDC callback server connection failed"); + } + }); + } +} + +fn handle_oidc_callback( + req: hyper::Request, + state: Arc, +) -> Response> { + if req.method() != Method::GET || !req.uri().path().starts_with("/callback") { + return Response::builder() + .status(StatusCode::NOT_FOUND) + .body(Full::new(Bytes::from("not found"))) + .expect("response"); + } + + let query = req.uri().query().unwrap_or(""); + let params: std::collections::HashMap = query + .split('&') + .filter_map(|pair| { + let mut parts = pair.splitn(2, '='); + let key = percent_decode(parts.next()?); + let value = percent_decode(parts.next().unwrap_or("")); + Some((key, value)) + }) + .collect(); + + // Check for error response from the IdP. + if let Some(error) = params.get("error") { + let desc = params.get("error_description").map_or("", String::as_str); + debug!(error = %error, description = %desc, "OIDC auth error"); + let _ = state.take_sender(); + return html_response( + StatusCode::BAD_REQUEST, + &format!("Authentication failed: {error}. {desc}"), + ); + } + + let code = match params.get("code") { + Some(c) if !c.is_empty() => c, + _ => { + let _ = state.take_sender(); + return html_response(StatusCode::BAD_REQUEST, "Missing authorization code."); + } + }; + + let received_state = params.get("state").map_or("", String::as_str); + if received_state != state.expected_state { + debug!("OIDC state mismatch"); + let _ = state.take_sender(); + return html_response(StatusCode::FORBIDDEN, "State parameter mismatch."); + } + + if let Some(sender) = state.take_sender() { + let _ = sender.send(code.clone()); + } + + html_response( + StatusCode::OK, + "Authentication successful! You can close this tab and return to the terminal.", + ) +} + +fn html_response(status: StatusCode, message: &str) -> Response> { + let body = format!( + "\ +

{message}

" + ); + Response::builder() + .status(status) + .header("content-type", "text/html") + .body(Full::new(Bytes::from(body))) + .expect("response") +} diff --git a/crates/openshell-cli/src/policy_update.rs b/crates/openshell-cli/src/policy_update.rs new file mode 100644 index 0000000000..322a28df67 --- /dev/null +++ b/crates/openshell-cli/src/policy_update.rs @@ -0,0 +1,481 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::collections::{BTreeMap, HashMap}; + +use miette::{Result, miette}; +use openshell_core::proto::policy_merge_operation; +use openshell_core::proto::{ + AddAllowRules, AddDenyRules, AddNetworkRule, L7Allow, L7DenyRule, L7Rule, NetworkBinary, + NetworkEndpoint, NetworkPolicyRule, PolicyMergeOperation, RemoveNetworkEndpoint, + RemoveNetworkRule, +}; +use openshell_policy::{PolicyMergeOp, generated_rule_name}; + +#[derive(Debug, Clone)] +pub struct PolicyUpdatePlan { + pub merge_operations: Vec, + pub preview_operations: Vec, +} + +pub fn build_policy_update_plan( + add_endpoints: &[String], + remove_endpoints: &[String], + add_deny: &[String], + add_allow: &[String], + remove_rules: &[String], + binaries: &[String], + rule_name: Option<&str>, +) -> Result { + if binaries.iter().any(|binary| binary.trim().is_empty()) { + return Err(miette!("--binary values must not be empty")); + } + if !binaries.is_empty() && add_endpoints.is_empty() { + return Err(miette!("--binary can only be used with --add-endpoint")); + } + if rule_name.is_some() && add_endpoints.is_empty() { + return Err(miette!("--rule-name can only be used with --add-endpoint")); + } + if rule_name.is_some() && add_endpoints.len() > 1 { + return Err(miette!( + "--rule-name is only supported when exactly one --add-endpoint is provided" + )); + } + + let mut merge_operations = Vec::new(); + let mut preview_operations = Vec::new(); + + let deduped_binaries = dedup_strings(binaries); + for spec in add_endpoints { + let endpoint = parse_add_endpoint_spec(spec)?; + let target_rule_name = rule_name + .map(str::trim) + .filter(|name| !name.is_empty()) + .map_or_else( + || generated_rule_name(&endpoint.host, endpoint.port), + ToString::to_string, + ); + let rule = NetworkPolicyRule { + name: target_rule_name.clone(), + endpoints: vec![endpoint.clone()], + binaries: deduped_binaries + .iter() + .map(|path| NetworkBinary { + path: path.clone(), + ..Default::default() + }) + .collect(), + }; + merge_operations.push(PolicyMergeOperation { + operation: Some(policy_merge_operation::Operation::AddRule(AddNetworkRule { + rule_name: target_rule_name.clone(), + rule: Some(rule.clone()), + })), + }); + preview_operations.push(PolicyMergeOp::AddRule { + rule_name: target_rule_name, + rule, + }); + } + + for spec in remove_endpoints { + let (host, port) = parse_remove_endpoint_spec(spec)?; + merge_operations.push(PolicyMergeOperation { + operation: Some(policy_merge_operation::Operation::RemoveEndpoint( + RemoveNetworkEndpoint { + rule_name: String::new(), + host: host.clone(), + port, + }, + )), + }); + preview_operations.push(PolicyMergeOp::RemoveEndpoint { + rule_name: None, + host, + port, + }); + } + + for name in remove_rules { + let rule_name = name.trim(); + if rule_name.is_empty() { + return Err(miette!("--remove-rule values must not be empty")); + } + merge_operations.push(PolicyMergeOperation { + operation: Some(policy_merge_operation::Operation::RemoveRule( + RemoveNetworkRule { + rule_name: rule_name.to_string(), + }, + )), + }); + preview_operations.push(PolicyMergeOp::RemoveRule { + rule_name: rule_name.to_string(), + }); + } + + for ((host, port), rules) in group_allow_rules(add_allow)? { + merge_operations.push(PolicyMergeOperation { + operation: Some(policy_merge_operation::Operation::AddAllowRules( + AddAllowRules { + host: host.clone(), + port, + rules: rules.clone(), + }, + )), + }); + preview_operations.push(PolicyMergeOp::AddAllowRules { host, port, rules }); + } + + for ((host, port), deny_rules) in group_deny_rules(add_deny)? { + merge_operations.push(PolicyMergeOperation { + operation: Some(policy_merge_operation::Operation::AddDenyRules( + AddDenyRules { + host: host.clone(), + port, + deny_rules: deny_rules.clone(), + }, + )), + }); + preview_operations.push(PolicyMergeOp::AddDenyRules { + host, + port, + deny_rules, + }); + } + + if merge_operations.is_empty() { + return Err(miette!( + "policy update requires at least one operation flag" + )); + } + + Ok(PolicyUpdatePlan { + merge_operations, + preview_operations, + }) +} + +fn group_allow_rules(specs: &[String]) -> Result>> { + let mut grouped = BTreeMap::new(); + for spec in specs { + let parsed = parse_l7_rule_spec("--add-allow", spec)?; + grouped + .entry((parsed.host, parsed.port)) + .or_insert_with(Vec::new) + .push(L7Rule { + allow: Some(L7Allow { + method: parsed.method, + path: parsed.path, + command: String::new(), + query: HashMap::default(), + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), + }), + }); + } + Ok(grouped) +} + +fn group_deny_rules(specs: &[String]) -> Result>> { + let mut grouped = BTreeMap::new(); + for spec in specs { + let parsed = parse_l7_rule_spec("--add-deny", spec)?; + grouped + .entry((parsed.host, parsed.port)) + .or_insert_with(Vec::new) + .push(L7DenyRule { + method: parsed.method, + path: parsed.path, + command: String::new(), + query: HashMap::default(), + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), + }); + } + Ok(grouped) +} + +#[derive(Debug, Clone)] +struct ParsedL7RuleSpec { + host: String, + port: u32, + method: String, + path: String, +} + +fn parse_l7_rule_spec(flag: &str, spec: &str) -> Result { + let parts = spec.split(':').collect::>(); + if parts.len() != 4 { + return Err(miette!( + "{flag} expects host:port:METHOD:path_glob, got '{spec}'" + )); + } + + let host = parse_host(flag, spec, parts[0])?; + let port = parse_port(flag, spec, parts[1])?; + let method = parts[2].trim(); + if method.is_empty() { + return Err(miette!("{flag} has an empty METHOD segment in '{spec}'")); + } + if method.contains(char::is_whitespace) { + return Err(miette!( + "{flag} METHOD must not contain whitespace in '{spec}'" + )); + } + + let path = parts[3].trim(); + if path.is_empty() { + return Err(miette!("{flag} has an empty path segment in '{spec}'")); + } + if !path.starts_with('/') && path != "**" && !path.starts_with("**/") { + return Err(miette!( + "{flag} path must start with '/' or be '**', got '{path}' in '{spec}'" + )); + } + + Ok(ParsedL7RuleSpec { + host, + port, + method: method.to_ascii_uppercase(), + path: path.to_string(), + }) +} + +fn parse_remove_endpoint_spec(spec: &str) -> Result<(String, u32)> { + let parts = spec.split(':').collect::>(); + if parts.len() != 2 { + return Err(miette!("--remove-endpoint expects host:port, got '{spec}'")); + } + + Ok(( + parse_host("--remove-endpoint", spec, parts[0])?, + parse_port("--remove-endpoint", spec, parts[1])?, + )) +} + +fn parse_add_endpoint_spec(spec: &str) -> Result { + let parts = spec.split(':').collect::>(); + if !(2..=5).contains(&parts.len()) { + return Err(miette!( + "--add-endpoint expects host:port[:access[:protocol[:enforcement]]], got '{spec}'" + )); + } + + let host = parse_host("--add-endpoint", spec, parts[0])?; + let port = parse_port("--add-endpoint", spec, parts[1])?; + + let access = parts.get(2).copied().unwrap_or("").trim(); + let protocol = parts.get(3).copied().unwrap_or("").trim(); + let enforcement = parts.get(4).copied().unwrap_or("").trim(); + + if parts.len() == 3 && access.is_empty() { + return Err(miette!( + "--add-endpoint has an empty access segment in '{spec}'; omit it entirely if you do not need access or protocol fields" + )); + } + if !enforcement.is_empty() && protocol.is_empty() { + return Err(miette!( + "--add-endpoint cannot set enforcement without protocol in '{spec}'" + )); + } + if !access.is_empty() && !matches!(access, "read-only" | "read-write" | "full") { + return Err(miette!( + "--add-endpoint access segment must be one of read-only, read-write, or full; got '{access}' in '{spec}'" + )); + } + if !protocol.is_empty() && !matches!(protocol, "rest" | "sql") { + return Err(miette!( + "--add-endpoint protocol segment must be 'rest' or 'sql'; got '{protocol}' in '{spec}'" + )); + } + if !enforcement.is_empty() && !matches!(enforcement, "enforce" | "audit") { + return Err(miette!( + "--add-endpoint enforcement segment must be 'enforce' or 'audit'; got '{enforcement}' in '{spec}'" + )); + } + + Ok(NetworkEndpoint { + host, + port, + ports: vec![port], + protocol: protocol.to_string(), + enforcement: enforcement.to_string(), + access: access.to_string(), + ..Default::default() + }) +} + +fn parse_host(flag: &str, spec: &str, host: &str) -> Result { + let host = host.trim(); + if host.is_empty() { + return Err(miette!("{flag} has an empty host segment in '{spec}'")); + } + if host.contains(char::is_whitespace) { + return Err(miette!( + "{flag} host must not contain whitespace in '{spec}'" + )); + } + if host.contains('/') { + return Err(miette!("{flag} host must not contain '/' in '{spec}'")); + } + Ok(host.to_string()) +} + +fn parse_port(flag: &str, spec: &str, port: &str) -> Result { + let port = port.trim(); + if port.is_empty() { + return Err(miette!("{flag} has an empty port segment in '{spec}'")); + } + let parsed = port.parse::().map_err(|_| { + miette!("{flag} port segment must be a base-10 integer, got '{port}' in '{spec}'") + })?; + if parsed == 0 || parsed > 65535 { + return Err(miette!( + "{flag} port must be in the range 1-65535, got '{parsed}' in '{spec}'" + )); + } + Ok(parsed) +} + +fn dedup_strings(values: &[String]) -> Vec { + let mut deduped = Vec::new(); + for value in values { + let trimmed = value.trim(); + if !trimmed.is_empty() && !deduped.iter().any(|existing| existing == trimmed) { + deduped.push(trimmed.to_string()); + } + } + deduped +} + +#[cfg(test)] +mod tests { + use super::build_policy_update_plan; + + #[test] + fn parse_add_endpoint_basic_l4() { + let plan = + build_policy_update_plan(&["ghcr.io:443".to_string()], &[], &[], &[], &[], &[], None) + .expect("plan should build"); + assert_eq!(plan.merge_operations.len(), 1); + assert_eq!(plan.preview_operations.len(), 1); + } + + #[test] + fn parse_add_endpoint_rejects_bad_access() { + let error = build_policy_update_plan( + &["api.github.com:443:write-ish".to_string()], + &[], + &[], + &[], + &[], + &[], + None, + ) + .expect_err("plan should fail"); + assert!(error.to_string().contains("access segment")); + } + + #[test] + fn parse_add_endpoint_allows_empty_access_when_protocol_present() { + build_policy_update_plan( + &["api.github.com:443::rest:enforce".to_string()], + &[], + &[], + &[], + &[], + &[], + None, + ) + .expect("plan should build"); + } + + #[test] + fn parse_add_deny_rejects_empty_method() { + let error = build_policy_update_plan( + &[], + &[], + &["api.github.com:443::/repos/**".to_string()], + &[], + &[], + &[], + None, + ) + .expect_err("plan should fail"); + assert!(error.to_string().contains("METHOD")); + } + + #[test] + fn parse_add_allow_rejects_non_absolute_path() { + let error = build_policy_update_plan( + &[], + &[], + &[], + &["api.github.com:443:GET:repos/**".to_string()], + &[], + &[], + None, + ) + .expect_err("plan should fail"); + assert!(error.to_string().contains("path must start with '/'")); + } + + #[test] + fn parse_add_endpoint_rejects_enforcement_without_protocol() { + let error = build_policy_update_plan( + &["api.github.com:443:read-only::enforce".to_string()], + &[], + &[], + &[], + &[], + &[], + None, + ) + .expect_err("plan should fail"); + assert!( + error + .to_string() + .contains("cannot set enforcement without protocol") + ); + } + + #[test] + fn parse_remove_endpoint_rejects_out_of_range_port() { + let error = build_policy_update_plan( + &[], + &["api.github.com:70000".to_string()], + &[], + &[], + &[], + &[], + None, + ) + .expect_err("plan should fail"); + assert!(error.to_string().contains("range 1-65535")); + } + + #[test] + fn binary_requires_add_endpoint() { + let error = + build_policy_update_plan(&[], &[], &[], &[], &[], &["/usr/bin/gh".to_string()], None) + .expect_err("plan should fail"); + assert!(error.to_string().contains("--binary")); + } + + #[test] + fn rule_name_rejects_multiple_add_endpoints() { + let error = build_policy_update_plan( + &["api.github.com:443".to_string(), "ghcr.io:443".to_string()], + &[], + &[], + &[], + &[], + &[], + Some("shared"), + ) + .expect_err("plan should fail"); + assert!(error.to_string().contains("exactly one --add-endpoint")); + } +} diff --git a/crates/openshell-cli/src/run.rs b/crates/openshell-cli/src/run.rs index c41b53518e..818fd3e8fb 100644 --- a/crates/openshell-cli/src/run.rs +++ b/crates/openshell-cli/src/run.rs @@ -3,8 +3,10 @@ //! CLI command implementations. +use crate::policy_update::build_policy_update_plan; use crate::tls::{ - TlsOptions, build_rustls_config, grpc_client, grpc_inference_client, require_tls_materials, + TlsOptions, build_insecure_rustls_config, build_rustls_config, grpc_client, + grpc_inference_client, require_tls_materials, }; use bytes::Bytes; use dialoguer::{Confirm, Select, theme::ColorfulTheme}; @@ -21,18 +23,21 @@ use openshell_bootstrap::{ get_gateway_metadata, list_gateways, load_active_gateway, remove_gateway_metadata, resolve_ssh_hostname, save_active_gateway, save_last_sandbox, store_gateway_metadata, }; +use openshell_core::proto::ProviderProfileCategory; use openshell_core::proto::{ ApproveAllDraftChunksRequest, ApproveDraftChunkRequest, ClearDraftChunksRequest, CreateProviderRequest, CreateSandboxRequest, DeleteProviderRequest, DeleteSandboxRequest, ExecSandboxRequest, GetClusterInferenceRequest, GetDraftHistoryRequest, GetDraftPolicyRequest, GetGatewayConfigRequest, GetProviderRequest, GetSandboxConfigRequest, GetSandboxLogsRequest, - GetSandboxPolicyStatusRequest, GetSandboxRequest, HealthRequest, ListProvidersRequest, - ListSandboxPoliciesRequest, ListSandboxesRequest, PolicyStatus, Provider, - RejectDraftChunkRequest, Sandbox, SandboxPhase, SandboxPolicy, SandboxSpec, SandboxTemplate, - SetClusterInferenceRequest, SettingScope, SettingValue, UpdateConfigRequest, - UpdateProviderRequest, WatchSandboxRequest, exec_sandbox_event, setting_value, + GetSandboxPolicyStatusRequest, GetSandboxRequest, HealthRequest, ListProviderProfilesRequest, + ListProvidersRequest, ListSandboxPoliciesRequest, ListSandboxesRequest, PolicySource, + PolicyStatus, Provider, ProviderProfile, RejectDraftChunkRequest, Sandbox, SandboxPhase, + SandboxPolicy, SandboxSpec, SandboxTemplate, SetClusterInferenceRequest, SettingScope, + SettingValue, UpdateConfigRequest, UpdateProviderRequest, WatchSandboxRequest, + exec_sandbox_event, setting_value, }; use openshell_core::settings::{self, SettingValueKind}; +use openshell_core::{ObjectId, ObjectName}; use openshell_providers::{ ProviderRegistry, detect_provider_from_command, normalize_provider_type, }; @@ -342,7 +347,7 @@ impl ProvisioningDisplay { } /// Clear all progress output (spinner, spacer, and completed step lines). - fn clear(&mut self) { + fn clear(&self) { self.spacer.finish_and_clear(); self.spinner.finish_and_clear(); for bar in &self.completed_bars { @@ -386,13 +391,16 @@ fn format_bytes(bytes: u64) -> String { const GB: u64 = 1024 * MB; if bytes >= GB { - format!("{:.1} GB", bytes as f64 / GB as f64) + // GB-scale precision loss is acceptable for a human-readable label. + #[allow(clippy::cast_precision_loss)] + let gb = bytes as f64 / GB as f64; + format!("{gb:.1} GB") } else if bytes >= MB { format!("{} MB", bytes / MB) } else if bytes >= KB { format!("{} KB", bytes / KB) } else { - format!("{} B", bytes) + format!("{bytes} B") } } @@ -402,7 +410,7 @@ fn print_sandbox_header(sandbox: &Sandbox, display: Option<&ProvisioningDisplay> format!( "{} {}", "Created sandbox:".cyan().bold(), - sandbox.name.bold() + sandbox.object_name().bold() ), String::new(), ]; @@ -424,9 +432,7 @@ const CLUSTER_DEPLOY_LOG_LINES: usize = 15; /// Return the current terminal width, falling back to 80 columns. fn term_width() -> usize { - crossterm::terminal::size() - .map(|(w, _)| w as usize) - .unwrap_or(80) + crossterm::terminal::size().map_or(80, |(w, _)| w as usize) } /// Build a horizontal rule of `─` characters with an optional centered label. @@ -441,7 +447,7 @@ fn horizontal_rule(label: Option<&str>, width: usize) -> String { let remaining = width - text_len; let left = remaining / 2; let right = remaining - left; - format!("{}{}{}", "─".repeat(left), text_with_pad, "─".repeat(right),) + format!("{}{}{}", "─".repeat(left), text_with_pad, "─".repeat(right)) } None => "─".repeat(width), } @@ -621,11 +627,10 @@ impl GatewayDeployLogPanel { } fn update_spinner_message(&self) { - let msg = if let Some(detail) = &self.progress { - format!("{} ({})", self.status, detail.dimmed()) - } else { - self.status.clone() - }; + let msg = self.progress.as_ref().map_or_else( + || self.status.clone(), + |detail| format!("{} ({})", self.status, detail.dimmed()), + ); self.spinner.set_message(msg); } @@ -863,6 +868,30 @@ fn is_loopback_gateway_endpoint(endpoint: &str) -> bool { } } +/// Check whether mTLS client certs exist on disk for the gateway that +/// would serve this endpoint. +/// +/// Loopback endpoints (`localhost`, `127.0.0.1`, `::1`) resolve to the +/// `"openshell"` gateway name, matching the convention used by +/// `init-pki.sh` and the TLS cert resolver in `tls.rs`. +fn mtls_certs_exist_for_endpoint(name: &str, endpoint: &str) -> bool { + let cert_name = if is_loopback_gateway_endpoint(endpoint) { + "openshell" + } else { + name + }; + openshell_core::paths::xdg_config_dir().is_ok_and(|d| { + let mtls = d + .join("openshell") + .join("gateways") + .join(cert_name) + .join("mtls"); + mtls.join("ca.crt").is_file() + && mtls.join("tls.crt").is_file() + && mtls.join("tls.key").is_file() + }) +} + fn plaintext_gateway_is_remote(endpoint: &str, remote: Option<&str>, local: bool) -> bool { if local { return false; @@ -879,13 +908,11 @@ fn plaintext_gateway_metadata( remote: Option<&str>, local: bool, ) -> GatewayMetadata { - let (remote_host, resolved_host) = if let Some(dest) = remote { + let (remote_host, resolved_host) = remote.map_or((None, None), |dest| { let ssh_host = extract_host_from_ssh_destination(dest); let resolved = resolve_ssh_hostname(&ssh_host); (Some(dest.to_string()), Some(resolved)) - } else { - (None, None) - }; + }); GatewayMetadata { name: name.to_string(), @@ -895,8 +922,8 @@ fn plaintext_gateway_metadata( remote_host, resolved_host, auth_mode: Some("plaintext".to_string()), - edge_team_domain: None, - edge_auth_url: None, + client_lifecycle_managed: Some(false), + ..Default::default() } } @@ -956,12 +983,17 @@ where /// /// An `ssh://` endpoint (e.g., `ssh://user@host:8080`) is shorthand for /// `--remote user@host` with the gateway endpoint derived from the URL. +#[allow(clippy::too_many_arguments)] pub async fn gateway_add( endpoint: &str, name: Option<&str>, remote: Option<&str>, ssh_key: Option<&str>, local: bool, + oidc_issuer: Option<&str>, + oidc_client_id: &str, + oidc_audience: Option<&str>, + oidc_scopes: Option<&str>, ) -> Result<()> { // If the endpoint starts with ssh://, parse it into an SSH destination // and a gateway endpoint automatically. The host is resolved via @@ -1020,9 +1052,14 @@ pub async fn gateway_add( } // Derive a gateway name from the hostname when none is provided. + // Loopback endpoints use the canonical "openshell" name, matching the + // convention in init-pki.sh, default_tls_dir, and bootstrap. let derived_name; let name = if let Some(n) = name { n + } else if is_loopback_gateway_endpoint(&endpoint) { + derived_name = "openshell".to_string(); + &derived_name } else { // Parse out just the host portion of the URL. derived_name = url::Url::parse(&endpoint) @@ -1043,7 +1080,111 @@ pub async fn gateway_add( )); } + // OIDC takes precedence over plaintext/mTLS/edge detection — the user + // explicitly opted in with --oidc-issuer regardless of scheme. + if let Some(issuer) = oidc_issuer { + // When --local is combined with --oidc-issuer, extract mTLS certs + // from the running container so the CLI can establish a TLS + // connection while using OIDC for application-level auth. + if local { + let endpoint_port = url::Url::parse(&endpoint).ok().and_then(|u| u.port()); + eprintln!("• Extracting TLS certificates from gateway container..."); + openshell_bootstrap::extract_and_store_pki(name, None, endpoint_port).await?; + } + + let metadata = GatewayMetadata { + name: name.to_string(), + gateway_endpoint: endpoint.clone(), + is_remote: !local, + auth_mode: Some("oidc".to_string()), + oidc_issuer: Some(issuer.to_string()), + oidc_client_id: Some(oidc_client_id.to_string()), + oidc_audience: oidc_audience.map(String::from), + oidc_scopes: oidc_scopes.map(String::from), + ..Default::default() + }; + + store_gateway_metadata(name, &metadata)?; + save_active_gateway(name)?; + + eprintln!( + "{} Gateway '{}' added and set as active", + "✓".green().bold(), + name, + ); + eprintln!(" {} {}", "Endpoint:".dimmed(), endpoint); + eprintln!(" {} oidc", "Auth:".dimmed()); + if local { + eprintln!("{} TLS certificates extracted", "✓".green().bold()); + } + eprintln!(); + + // Check for client_credentials env var (CI mode). + if std::env::var("OPENSHELL_OIDC_CLIENT_SECRET").is_ok() { + match crate::oidc_auth::oidc_client_credentials_flow( + issuer, + oidc_client_id, + oidc_audience, + oidc_scopes, + ) + .await + { + Ok(bundle) => { + openshell_bootstrap::oidc_token::store_oidc_token(name, &bundle)?; + eprintln!( + "{} Authenticated via client credentials", + "✓".green().bold() + ); + } + Err(e) => { + eprintln!("{} Authentication failed: {e}", "!".yellow()); + } + } + } else { + match crate::oidc_auth::oidc_browser_auth_flow( + issuer, + oidc_client_id, + oidc_audience, + oidc_scopes, + ) + .await + { + Ok(bundle) => { + openshell_bootstrap::oidc_token::store_oidc_token(name, &bundle)?; + eprintln!("{} Authenticated successfully", "✓".green().bold()); + } + Err(e) => { + eprintln!("{} Authentication skipped: {e}", "!".yellow()); + eprintln!( + " Authenticate later with: {}", + "openshell gateway login".dimmed(), + ); + } + } + } + + return Ok(()); + } + if endpoint.starts_with("http://") { + // Warn if mTLS certs exist for this gateway — the user likely + // meant to use https:// instead of http://. + let has_mtls_certs = mtls_certs_exist_for_endpoint(name, &endpoint); + + if has_mtls_certs { + let https_endpoint = endpoint.replacen("http://", "https://", 1); + let suggestion = if is_loopback_gateway_endpoint(&endpoint) { + format!("openshell gateway add --local {https_endpoint}") + } else { + format!("openshell gateway add {https_endpoint}") + }; + eprintln!( + "{} mTLS certificates found for gateway '{name}'. Did you mean to use https?", + "⚠".yellow().bold(), + ); + eprintln!(" Try: {suggestion}"); + } + let metadata = plaintext_gateway_metadata(name, &endpoint, remote, local); let gateway_type = gateway_type_label(&metadata); let gateway_auth = gateway_auth_label(&metadata); @@ -1051,6 +1192,21 @@ pub async fn gateway_add( store_gateway_metadata(name, &metadata)?; save_active_gateway(name)?; + // Verify the gateway is reachable. + let tls = TlsOptions::default(); + match http_health_check(&endpoint, &tls).await { + Ok(Some(status)) if status.is_success() => {} + _ => { + eprintln!( + "{} Gateway is not reachable at {endpoint}", + "⚠".yellow().bold(), + ); + if !has_mtls_certs { + eprintln!(" Verify the gateway is running and the endpoint is correct."); + } + } + } + eprintln!( "{} Gateway '{}' added and set as active", "✓".green().bold(), @@ -1077,18 +1233,26 @@ pub async fn gateway_add( // is not registered. Pass the endpoint port so the container can be // identified by its host port binding when multiple gateways run on // the same Docker host. - let endpoint_port = url::Url::parse(&endpoint).ok().and_then(|u| u.port()); - eprintln!("• Extracting TLS certificates from gateway container..."); - openshell_bootstrap::extract_and_store_pki(name, remote_opts.as_ref(), endpoint_port) - .await?; + // + // Skip extraction when client certs are already on disk (e.g., + // RPM/systemd deployments where init-pki.sh pre-provisions them + // before the gateway starts). + let certs_on_disk = mtls_certs_exist_for_endpoint(name, &endpoint); - let (remote_host, resolved_host) = if let Some(dest) = remote { + if certs_on_disk { + eprintln!("• TLS certificates already present, skipping extraction"); + } else { + let endpoint_port = url::Url::parse(&endpoint).ok().and_then(|u| u.port()); + eprintln!("• Extracting TLS certificates from gateway container..."); + openshell_bootstrap::extract_and_store_pki(name, remote_opts.as_ref(), endpoint_port) + .await?; + } + + let (remote_host, resolved_host) = remote.map_or((None, None), |dest| { let ssh_host = extract_host_from_ssh_destination(dest); let resolved = resolve_ssh_hostname(&ssh_host); (Some(dest.to_string()), Some(resolved)) - } else { - (None, None) - }; + }); let metadata = GatewayMetadata { name: name.to_string(), @@ -1098,13 +1262,25 @@ pub async fn gateway_add( remote_host, resolved_host, auth_mode: Some("mtls".to_string()), - edge_team_domain: None, - edge_auth_url: None, + client_lifecycle_managed: Some(false), + ..Default::default() }; store_gateway_metadata(name, &metadata)?; save_active_gateway(name)?; + // Verify the gateway is reachable over mTLS. + let tls = TlsOptions::default().with_gateway_name(name); + match http_health_check(&endpoint, &tls).await { + Ok(Some(status)) if status.is_success() => {} + _ => { + eprintln!( + "{} Gateway is not reachable at {endpoint}. Verify the gateway is running.", + "⚠".yellow().bold(), + ); + } + } + eprintln!( "{} Gateway '{}' added and set as active", "✓".green().bold(), @@ -1116,19 +1292,24 @@ pub async fn gateway_add( "Type:".dimmed(), if local { "local" } else { "remote" }, ); - eprintln!("{} TLS certificates extracted", "✓".green().bold()); + eprintln!( + "{} TLS certificates {}", + "✓".green().bold(), + if certs_on_disk { + "already present" + } else { + "extracted" + } + ); } else { // Cloud (edge-authenticated) gateway. let metadata = GatewayMetadata { name: name.to_string(), gateway_endpoint: endpoint.clone(), is_remote: true, - gateway_port: 0, - remote_host: None, - resolved_host: None, auth_mode: Some("cloudflare_jwt".to_string()), - edge_team_domain: None, - edge_auth_url: None, + client_lifecycle_managed: Some(false), + ..Default::default() }; store_gateway_metadata(name, &metadata)?; @@ -1161,9 +1342,9 @@ pub async fn gateway_add( Ok(()) } -/// Re-authenticate with an edge-authenticated gateway. +/// Re-authenticate with an edge-authenticated or OIDC gateway. /// -/// Opens a browser for edge proxy login and stores the updated token. +/// Dispatches to the appropriate auth flow based on `auth_mode`. pub async fn gateway_login(name: &str) -> Result<()> { let metadata = openshell_bootstrap::load_gateway_metadata(name).map_err(|_| { miette::miette!( @@ -1172,18 +1353,91 @@ pub async fn gateway_login(name: &str) -> Result<()> { ) })?; - if metadata.auth_mode.as_deref() != Some("cloudflare_jwt") { - return Err(miette::miette!( - "Gateway '{name}' does not use edge authentication.\n\ - Only edge-authenticated gateways support browser login." - )); + match metadata.auth_mode.as_deref() { + Some("cloudflare_jwt") => { + let token = crate::auth::browser_auth_flow(&metadata.gateway_endpoint).await?; + openshell_bootstrap::edge_token::store_edge_token(name, &token)?; + eprintln!("{} Authenticated to gateway '{name}'", "✓".green().bold()); + } + Some("oidc") => { + let issuer = metadata.oidc_issuer.as_deref().ok_or_else(|| { + miette::miette!("Gateway '{name}' has OIDC auth but no issuer URL in metadata") + })?; + let client_id = metadata + .oidc_client_id + .as_deref() + .unwrap_or("openshell-cli"); + let audience = metadata.oidc_audience.as_deref(); + let scopes = metadata.oidc_scopes.as_deref(); + + let bundle = if std::env::var("OPENSHELL_OIDC_CLIENT_SECRET").is_ok() { + crate::oidc_auth::oidc_client_credentials_flow(issuer, client_id, audience, scopes) + .await? + } else { + crate::oidc_auth::oidc_browser_auth_flow(issuer, client_id, audience, scopes) + .await? + }; + + let username = jwt_preferred_username(&bundle.access_token); + openshell_bootstrap::oidc_token::store_oidc_token(name, &bundle)?; + + if let Some(user) = username { + eprintln!( + "{} Authenticated to gateway '{name}' as {user}", + "✓".green().bold(), + ); + } else { + eprintln!("{} Authenticated to gateway '{name}'", "✓".green().bold()); + } + } + _ => { + return Err(miette::miette!( + "Gateway '{name}' does not use edge or OIDC authentication.\n\ + Only edge-authenticated and OIDC gateways support browser login." + )); + } } - let token = crate::auth::browser_auth_flow(&metadata.gateway_endpoint).await?; - openshell_bootstrap::edge_token::store_edge_token(name, &token)?; + Ok(()) +} - eprintln!("{} Authenticated to gateway '{name}'", "✓".green().bold(),); +/// Extract `preferred_username` from a JWT payload without signature verification. +fn jwt_preferred_username(token: &str) -> Option { + let payload = token.split('.').nth(1)?; + let decoded = + base64::Engine::decode(&base64::engine::general_purpose::URL_SAFE_NO_PAD, payload).ok()?; + let claims: serde_json::Value = serde_json::from_slice(&decoded).ok()?; + claims + .get("preferred_username") + .and_then(|v| v.as_str()) + .map(String::from) +} +/// Clear stored authentication credentials for a gateway. +pub fn gateway_logout(name: &str) -> Result<()> { + let metadata = openshell_bootstrap::load_gateway_metadata(name).map_err(|_| { + miette::miette!( + "Unknown gateway '{name}'.\n\ + List available gateways: openshell gateway select" + ) + })?; + + match metadata.auth_mode.as_deref() { + Some("oidc") => { + openshell_bootstrap::oidc_token::remove_oidc_token(name)?; + } + Some("cloudflare_jwt") => { + openshell_bootstrap::edge_token::remove_edge_token(name)?; + } + _ => { + return Err(miette::miette!( + "Gateway '{name}' uses {} authentication — no stored credentials to clear.", + metadata.auth_mode.as_deref().unwrap_or("mtls") + )); + } + } + + eprintln!("{} Logged out of gateway '{name}'", "✓".green().bold()); Ok(()) } @@ -1256,7 +1510,14 @@ async fn http_health_check(server: &str, tls: &TlsOptions) -> Result, @@ -1434,6 +1696,14 @@ pub async fn gateway_admin_deploy( registry_username: Option<&str>, registry_token: Option<&str>, gpu: Vec, + oidc_issuer: Option<&str>, + oidc_audience: &str, + oidc_client_id: &str, + oidc_roles_claim: Option<&str>, + oidc_admin_role: Option<&str>, + oidc_user_role: Option<&str>, + oidc_scopes: Option<&str>, + oidc_scopes_claim: Option<&str>, ) -> Result<()> { let location = if remote.is_some() { "remote" } else { "local" }; @@ -1448,18 +1718,16 @@ pub async fn gateway_admin_deploy( }); // If the gateway is already running and we're not recreating, short-circuit. - if !recreate { - if let Some(existing) = + if !recreate + && let Some(existing) = openshell_bootstrap::check_existing_deployment(name, remote_opts.as_ref()).await? - { - if existing.container_running { - eprintln!( - "{} Gateway '{name}' is already running.", - "✓".green().bold() - ); - return Ok(()); - } - } + && existing.container_running + { + eprintln!( + "{} Gateway '{name}' is already running.", + "✓".green().bold() + ); + return Ok(()); } // When resuming an existing gateway (not recreating), prefer the port @@ -1467,10 +1735,10 @@ pub async fn gateway_admin_deploy( // may have originally bootstrapped on a non-default port (e.g. `--port // 8082`) or with `--gateway-host host.docker.internal`, and a bare // `gateway start` without those flags should honour the original values. - let stored_metadata = if !recreate { - openshell_bootstrap::load_gateway_metadata(name).ok() - } else { + let stored_metadata = if recreate { None + } else { + openshell_bootstrap::load_gateway_metadata(name).ok() }; let effective_port = stored_metadata .as_ref() @@ -1500,8 +1768,34 @@ pub async fn gateway_admin_deploy( if let Some(token) = registry_token { options = options.with_registry_token(token); } + if let Some(issuer) = oidc_issuer { + options = options.with_oidc_issuer(issuer); + options = options.with_oidc_audience(oidc_audience); + options.oidc_client_id = oidc_client_id.to_string(); + if let Some(claim) = oidc_roles_claim { + options.oidc_roles_claim = Some(claim.to_string()); + } + if let Some(role) = oidc_admin_role { + options.oidc_admin_role = Some(role.to_string()); + } + if let Some(role) = oidc_user_role { + options.oidc_user_role = Some(role.to_string()); + } + if let Some(claim) = oidc_scopes_claim { + options.oidc_scopes_claim = Some(claim.to_string()); + } + } + + let handle = Box::pin(deploy_gateway_with_panel(options, name, location)).await?; - let handle = deploy_gateway_with_panel(options, name, location).await?; + // Persist oidc_scopes in gateway metadata so `gateway login` can + // request the correct scopes later. + if let Some(scopes) = oidc_scopes + && let Ok(mut meta) = openshell_bootstrap::load_gateway_metadata(name) + { + meta.oidc_scopes = Some(scopes.to_string()); + let _ = store_gateway_metadata(name, &meta); + } // Wait for the gRPC endpoint to actually accept connections before // declaring the gateway ready. The Docker health check may pass before @@ -1547,10 +1841,20 @@ fn resolve_gateway_control_target_from( } match metadata { - Some(metadata) if metadata.is_remote => metadata.remote_host.map_or( + // Not client-managed (`gateway add`) — the gateway lifecycle is + // managed externally (e.g. systemd, Podman, bare-metal); only + // remove the local registration metadata on destroy/stop. + Some(ref m) if m.client_lifecycle_managed == Some(false) => { + GatewayControlTarget::ExternalRegistration + } + // Remote gateway with SSH destination — managed remote container. + Some(ref m) if m.is_remote => m.remote_host.clone().map_or( GatewayControlTarget::ExternalRegistration, GatewayControlTarget::Remote, ), + // Client-managed (`gateway start`) or legacy metadata (no + // `client_lifecycle_managed` field) — treat as a + // locally-managed container. _ => GatewayControlTarget::Local, } } @@ -1686,6 +1990,7 @@ pub fn gateway_admin_info(name: &str) -> Result<()> { /// /// Connects to the Docker daemon (local or remote via SSH) and retrieves /// logs from the `openshell-cluster-{name}` container. +#[allow(clippy::future_not_send)] // Holds stdout lock; CLI command, never sent across threads. pub async fn doctor_logs( name: &str, lines: Option, @@ -1694,24 +1999,29 @@ pub async fn doctor_logs( ssh_key: Option<&str>, ) -> Result<()> { // Build remote options: explicit --remote flag, or auto-resolve from metadata - let remote_opts = if let Some(dest) = remote { - let mut opts = RemoteOptions::new(dest); - if let Some(key) = ssh_key { - opts = opts.with_ssh_key(key); - } - Some(opts) - } else if let Some(metadata) = get_gateway_metadata(name) - && metadata.is_remote - && let Some(ref host) = metadata.remote_host - { - let mut opts = RemoteOptions::new(host.clone()); - if let Some(key) = ssh_key { - opts = opts.with_ssh_key(key); - } - Some(opts) - } else { - None - }; + let remote_opts = remote.map_or_else( + || { + if let Some(metadata) = get_gateway_metadata(name) + && metadata.is_remote + && let Some(ref host) = metadata.remote_host + { + let mut opts = RemoteOptions::new(host.clone()); + if let Some(key) = ssh_key { + opts = opts.with_ssh_key(key); + } + Some(opts) + } else { + None + } + }, + |dest| { + let mut opts = RemoteOptions::new(dest); + if let Some(key) = ssh_key { + opts = opts.with_ssh_key(key); + } + Some(opts) + }, + ); let stdout = std::io::stdout().lock(); openshell_bootstrap::gateway_container_logs(remote_opts.as_ref(), name, lines, tail, stdout) @@ -1742,15 +2052,18 @@ pub fn doctor_exec( }; // Resolve remote destination: explicit --remote flag, or auto-resolve from metadata - let remote_host = if let Some(dest) = remote { - Some(dest.to_string()) - } else if let Some(metadata) = get_gateway_metadata(name) - && metadata.is_remote - { - metadata.remote_host.clone() - } else { - None - }; + let remote_host = remote.map_or_else( + || { + if let Some(metadata) = get_gateway_metadata(name) + && metadata.is_remote + { + metadata.remote_host + } else { + None + } + }, + |dest| Some(dest.to_string()), + ); let mut cmd = if let Some(ref host) = remote_host { validate_ssh_host(host)?; @@ -1826,6 +2139,7 @@ pub fn doctor_llm() -> Result<()> { /// /// Checks Docker connectivity and reports the result. Returns exit code 0 /// if all checks pass, 1 otherwise. +#[allow(clippy::future_not_send)] // Holds stdout lock; CLI command, never sent across threads. pub async fn doctor_check() -> Result<()> { use std::io::Write; let mut stdout = std::io::stdout().lock(); @@ -1846,7 +2160,7 @@ pub async fn doctor_check() -> Result<()> { match std::env::var("DOCKER_HOST") { Ok(val) => writeln!(stdout, "{val}").into_diagnostic()?, Err(_) => writeln!(stdout, "(not set, using default socket)").into_diagnostic()?, - }; + } writeln!(stdout, "\nAll checks passed.").into_diagnostic()?; Ok(()) @@ -1921,6 +2235,7 @@ pub async fn sandbox_create_with_bootstrap( upload: Option<&(String, Option, bool)>, keep: bool, gpu: bool, + gpu_device: Option<&str>, editor: Option, remote: Option<&str>, ssh_key: Option<&str>, @@ -1940,11 +2255,15 @@ pub async fn sandbox_create_with_bootstrap( )); } let requested_gpu = gpu || from.is_some_and(source_requests_gpu); - let (tls, server, gateway_name) = - crate::bootstrap::run_bootstrap(remote, ssh_key, requested_gpu).await?; + let (tls, server, gateway_name) = Box::pin(crate::bootstrap::run_bootstrap( + remote, + ssh_key, + requested_gpu, + )) + .await?; // Disable bootstrap inside sandbox_create so that a transient connection // failure right after deploy does not trigger a second bootstrap attempt. - sandbox_create( + Box::pin(sandbox_create( &server, name, from, @@ -1952,6 +2271,7 @@ pub async fn sandbox_create_with_bootstrap( upload, keep, gpu, + gpu_device, editor, remote, ssh_key, @@ -1962,8 +2282,9 @@ pub async fn sandbox_create_with_bootstrap( tty_override, Some(false), auto_providers_override, + &HashMap::new(), &tls, - ) + )) .await } @@ -1998,7 +2319,7 @@ async fn finalize_sandbox_create_session( } /// Create a sandbox with default settings. -#[allow(clippy::too_many_arguments)] +#[allow(clippy::too_many_arguments, clippy::implicit_hasher)] // user-facing CLI command; default hasher is fine pub async fn sandbox_create( server: &str, name: Option<&str>, @@ -2007,6 +2328,7 @@ pub async fn sandbox_create( upload: Option<&(String, Option, bool)>, keep: bool, gpu: bool, + gpu_device: Option<&str>, editor: Option, remote: Option<&str>, ssh_key: Option<&str>, @@ -2017,6 +2339,7 @@ pub async fn sandbox_create( tty_override: Option, bootstrap_override: Option, auto_providers_override: Option, + labels: &HashMap, tls: &TlsOptions, ) -> Result<()> { if editor.is_some() && !command.is_empty() { @@ -2065,8 +2388,12 @@ pub async fn sandbox_create( return Err(err); } let requested_gpu = gpu || from.is_some_and(source_requests_gpu); - let (new_tls, new_server, _) = - crate::bootstrap::run_bootstrap(remote, ssh_key, requested_gpu).await?; + let (new_tls, new_server, _) = Box::pin(crate::bootstrap::run_bootstrap( + remote, + ssh_key, + requested_gpu, + )) + .await?; let c = grpc_client(&new_server, &new_tls) .await .wrap_err("bootstrap succeeded but failed to connect to gateway")?; @@ -2113,12 +2440,14 @@ pub async fn sandbox_create( let request = CreateSandboxRequest { spec: Some(SandboxSpec { gpu: requested_gpu, + gpu_device: gpu_device.unwrap_or_default().to_string(), policy, providers: configured_providers, template, ..SandboxSpec::default() }), name: name.unwrap_or_default().to_string(), + labels: labels.clone(), }; let response = match client.create_sandbox(request).await { @@ -2138,7 +2467,11 @@ pub async fn sandbox_create( let interactive = std::io::stdout().is_terminal(); let persist = sandbox_should_persist(keep, forward.as_ref()); - let sandbox_name = sandbox.name.clone(); + let sandbox_name = if sandbox.object_name().is_empty() { + "unknown".to_string() + } else { + sandbox.object_name().to_string() + }; // Record this sandbox as the last-used for the active gateway only when it // is expected to persist beyond the initial session. @@ -2173,9 +2506,14 @@ pub async fn sandbox_create( // a newly created sandbox. Instead we handle termination client-side: // we wait until we have observed at least one non-Ready phase followed // by Ready (a genuine Provisioning → Ready transition). + let sandbox_id = if sandbox.object_id().is_empty() { + "unknown".to_string() + } else { + sandbox.object_id().to_string() + }; let mut stream = client .watch_sandbox(WatchSandboxRequest { - id: sandbox.id.clone(), + id: sandbox_id.clone(), follow_status: true, follow_logs: true, follow_events: true, @@ -2318,7 +2656,7 @@ pub async fn sandbox_create( let label = if size_label.is_empty() { "Image pulled".to_string() } else { - format!("Image pulled ({})", size_label) + format!("Image pulled ({size_label})") }; if let Some(d) = display.as_mut() { d.complete_step_with_label( @@ -2357,10 +2695,10 @@ pub async fn sandbox_create( eprintln!(" {} {} {}", ts.dimmed(), "WARN".yellow(), w.message); } } - Some(openshell_core::proto::sandbox_stream_event::Payload::DraftPolicyUpdate(_)) => { + Some(openshell_core::proto::sandbox_stream_event::Payload::DraftPolicyUpdate(_)) + | None => { // Draft policy updates are handled in the draft panel, not during provisioning. } - None => {} } } @@ -2403,6 +2741,7 @@ pub async fn sandbox_create( &sandbox_name, &base_dir, &files, + local, dest, &effective_tls, ) @@ -2417,7 +2756,7 @@ pub async fn sandbox_create( ) .await?; } - eprintln!(" {} Files uploaded", "\u{2713}".green().bold(),); + eprintln!(" {} Files uploaded", "\u{2713}".green().bold()); } // If --forward was requested, start the background port forward @@ -2534,6 +2873,7 @@ pub async fn sandbox_create( } /// Resolved source for the `--from` flag on `sandbox create`. +#[derive(Debug)] enum ResolvedSource { /// A ready-to-use container image reference. Image(String), @@ -2550,19 +2890,15 @@ enum ResolvedSource { /// Resolution order: /// 1. Existing file whose name contains "Dockerfile" → build from file. /// 2. Existing directory that contains a `Dockerfile` → build from directory. -/// 3. Value contains `/`, `:`, or `.` → treat as a full image reference. -/// 4. Otherwise → community sandbox name, expanded via the registry prefix. +/// 3. Missing explicit local paths → local error, not image pull. +/// 4. Value contains `/`, `:`, or `.` → treat as a full image reference. +/// 5. Otherwise → community sandbox name, expanded via the registry prefix. fn resolve_from(value: &str) -> Result { let path = Path::new(value); // 1. Existing file that looks like a Dockerfile. if path.is_file() { - let name = path - .file_name() - .map(|n| n.to_string_lossy()) - .unwrap_or_default(); - let lower = name.to_lowercase(); - if lower.contains("dockerfile") || lower.ends_with(".dockerfile") { + if filename_looks_like_dockerfile(path) { let dockerfile = path .canonicalize() .into_diagnostic() @@ -2576,6 +2912,13 @@ fn resolve_from(value: &str) -> Result { context, }); } + + if value_looks_like_local_source(value) { + return Err(miette::miette!( + "local --from file is not a Dockerfile: {}", + path.display() + )); + } } // 2. Existing directory containing a Dockerfile. @@ -2598,22 +2941,62 @@ fn resolve_from(value: &str) -> Result { )); } - // 3. Full image reference or community sandbox name — delegate to shared + if path.exists() { + return Err(miette::miette!( + "local --from path is not a regular file or directory: {}", + path.display() + )); + } + + // 3. Missing explicit local paths should fail locally. Otherwise values + // like `./Dockerfile` reach the gateway as image references and fail as + // Docker pull errors. + if value_looks_like_local_source(value) { + return Err(miette::miette!( + "local --from path does not exist: {}\n\ + Use an existing Dockerfile, a directory containing Dockerfile, or a container image reference.", + path.display() + )); + } + + // 4. Full image reference or community sandbox name — delegate to shared // resolution in openshell-core. Ok(ResolvedSource::Image( openshell_core::image::resolve_community_image(value), )) } +fn filename_looks_like_dockerfile(path: &Path) -> bool { + let name = path + .file_name() + .map(|n| n.to_string_lossy()) + .unwrap_or_default(); + let lower = name.to_lowercase(); + lower.contains("dockerfile") || lower.ends_with(".dockerfile") +} + +fn value_looks_like_local_source(value: &str) -> bool { + value_is_explicit_local_path(value) || value_looks_like_bare_dockerfile_name(value) +} + +fn value_is_explicit_local_path(value: &str) -> bool { + let path = Path::new(value); + path.is_absolute() + || matches!(value, "." | "..") + || value.starts_with("./") + || value.starts_with("../") + || value.starts_with("~/") +} + +fn value_looks_like_bare_dockerfile_name(value: &str) -> bool { + !value.contains('/') && !value.contains(':') && filename_looks_like_dockerfile(Path::new(value)) +} + fn source_requests_gpu(source: &str) -> bool { - if let Ok(resolved) = resolve_from(source) { - match resolved { - ResolvedSource::Image(image) => image_requests_gpu(&image), - ResolvedSource::Dockerfile { .. } => false, - } - } else { - false - } + resolve_from(source).is_ok_and(|resolved| match resolved { + ResolvedSource::Image(image) => image_requests_gpu(&image), + ResolvedSource::Dockerfile { .. } => false, + }) } fn image_requests_gpu(image: &str) -> bool { @@ -2629,15 +3012,29 @@ fn image_requests_gpu(image: &str) -> bool { image_name.contains("gpu") } -/// Build a Dockerfile and push the resulting image into the gateway. +fn dockerfile_sources_supported_for_gateway(metadata: Option<&GatewayMetadata>) -> bool { + !metadata.is_some_and(|metadata| metadata.is_remote) +} + +/// Build a Dockerfile and make the resulting image available to the gateway. /// -/// Returns the image tag that was built so the caller can use it for sandbox -/// creation. +/// For local Kubernetes gateways running in Docker, this imports the built image +/// into the gateway runtime and returns the Docker tag. Standalone local +/// gateways use the same Docker daemon that the CLI built into, so the tag is +/// passed through directly and the active compute driver resolves it. async fn build_from_dockerfile( dockerfile: &Path, context: &Path, gateway_name: &str, ) -> Result { + let metadata = get_gateway_metadata(gateway_name); + if !dockerfile_sources_supported_for_gateway(metadata.as_ref()) { + return Err(miette!( + "local Dockerfile sources are only supported for local gateways; gateway '{}' is remote", + gateway_name + )); + } + let timestamp = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .unwrap_or_default() @@ -2657,21 +3054,39 @@ async fn build_from_dockerfile( eprintln!(" {msg}"); }; - openshell_bootstrap::build::build_and_push_image( + openshell_bootstrap::build::build_local_image( dockerfile, &tag, context, - gateway_name, &HashMap::new(), &mut on_log, ) .await?; + let existing_gateway = openshell_bootstrap::check_existing_deployment(gateway_name, None) + .await + .wrap_err("failed to inspect local gateway deployment state")?; + let pushed_into_gateway = existing_gateway + .is_some_and(|gateway| gateway.container_exists && gateway.container_running); + if pushed_into_gateway { + openshell_bootstrap::build::push_image_into_gateway(&tag, gateway_name, &mut on_log) + .await?; + eprintln!(); + eprintln!( + "{} Image {} is available in the gateway.", + "✓".green().bold(), + tag.cyan(), + ); + eprintln!(); + return Ok(tag); + } + eprintln!(); eprintln!( - "{} Image {} is available in the gateway.", + "{} Image {} is available in the local Docker daemon for gateway '{}'.", "✓".green().bold(), tag.cyan(), + gateway_name, ); eprintln!(); @@ -2733,7 +3148,16 @@ pub async fn sandbox_sync_command( } /// Fetch a sandbox by name. -pub async fn sandbox_get(server: &str, name: &str, tls: &TlsOptions) -> Result<()> { +/// +/// Policy always comes from [`GetSandboxConfig`] (effective active policy, sandbox +/// or global). With `policy_only`, prints only that YAML to stdout; otherwise +/// prints sandbox metadata and the same policy with formatted YAML. +pub async fn sandbox_get( + server: &str, + name: &str, + policy_only: bool, + tls: &TlsOptions, +) -> Result<()> { let mut client = grpc_client(server, tls).await?; let response = client @@ -2747,16 +3171,86 @@ pub async fn sandbox_get(server: &str, name: &str, tls: &TlsOptions) -> Result<( .sandbox .ok_or_else(|| miette::miette!("sandbox missing from response"))?; + let sandbox_id = if sandbox.object_id().is_empty() { + return Err(miette::miette!("sandbox missing metadata")); + } else { + sandbox.object_id().to_string() + }; + + let config = client + .get_sandbox_config(GetSandboxConfigRequest { sandbox_id }) + .await + .into_diagnostic()? + .into_inner(); + + if policy_only { + let Some(ref policy) = config.policy else { + return Err(miette::miette!( + "no active policy configured for this sandbox" + )); + }; + let yaml_str = openshell_policy::serialize_sandbox_policy(policy) + .wrap_err("failed to serialize policy to YAML")?; + print!("{yaml_str}"); + return Ok(()); + } + println!("{}", "Sandbox:".cyan().bold()); println!(); - println!(" {} {}", "Id:".dimmed(), sandbox.id); - println!(" {} {}", "Name:".dimmed(), sandbox.name); - println!(" {} {}", "Namespace:".dimmed(), sandbox.namespace); + let id = if sandbox.object_id().is_empty() { + "unknown" + } else { + sandbox.object_id() + }; + let name = if sandbox.object_name().is_empty() { + "unknown" + } else { + sandbox.object_name() + }; + println!(" {} {}", "Id:".dimmed(), id); + println!(" {} {}", "Name:".dimmed(), name); println!(" {} {}", "Phase:".dimmed(), phase_name(sandbox.phase)); - if let Some(spec) = &sandbox.spec - && let Some(policy) = &spec.policy + // Display labels if present + if let Some(metadata) = &sandbox.metadata + && !metadata.labels.is_empty() { + println!(" {} ", "Labels:".dimmed()); + let mut labels: Vec<_> = metadata.labels.iter().collect(); + labels.sort_by_key(|(k, _)| *k); + for (key, value) in labels { + println!(" {key}: {value}"); + } + } + + let policy_from_global = config.policy_source == PolicySource::Global as i32; + println!( + " {} {}", + "Policy source:".dimmed(), + if policy_from_global { + "global" + } else { + "sandbox" + } + ); + let revision = if policy_from_global { + if config.global_policy_version > 0 { + Some(config.global_policy_version) + } else if config.version > 0 { + Some(config.version) + } else { + None + } + } else if config.version > 0 { + Some(config.version) + } else { + None + }; + if let Some(rev) = revision { + println!(" {} {}", "Revision:".dimmed(), rev); + } + + if let Some(ref policy) = config.policy { println!(); print_sandbox_policy(policy); } @@ -2805,7 +3299,9 @@ pub async fn sandbox_exec_grpc( // Read stdin if piped (not a TTY), using spawn_blocking to avoid blocking // the async runtime. Cap the read at MAX_STDIN_PAYLOAD + 1 so we never // buffer more than the limit into memory. - let stdin_payload = if !std::io::stdin().is_terminal() { + let stdin_payload = if std::io::stdin().is_terminal() { + Vec::new() + } else { tokio::task::spawn_blocking(|| { let limit = (MAX_STDIN_PAYLOAD + 1) as u64; let mut buf = Vec::new(); @@ -2823,8 +3319,6 @@ pub async fn sandbox_exec_grpc( }) .await .into_diagnostic()?? // first ? unwraps JoinError, second ? unwraps Result - } else { - Vec::new() }; // Resolve TTY mode: explicit --tty / --no-tty wins, otherwise auto-detect. @@ -2834,7 +3328,7 @@ pub async fn sandbox_exec_grpc( // Make the streaming gRPC call. let mut stream = client .exec_sandbox(ExecSandboxRequest { - sandbox_id: sandbox.id, + sandbox_id: sandbox.object_id().to_string(), command: command.to_vec(), workdir: workdir.unwrap_or_default().to_string(), environment: HashMap::new(), @@ -2938,12 +3432,17 @@ pub async fn sandbox_list( offset: u32, ids_only: bool, names_only: bool, + label_selector: Option<&str>, tls: &TlsOptions, ) -> Result<()> { let mut client = grpc_client(server, tls).await?; let response = client - .list_sandboxes(ListSandboxesRequest { limit, offset }) + .list_sandboxes(ListSandboxesRequest { + limit, + offset, + label_selector: label_selector.unwrap_or("").to_string(), + }) .await .into_diagnostic()?; @@ -2957,14 +3456,14 @@ pub async fn sandbox_list( if ids_only { for sandbox in sandboxes { - println!("{}", sandbox.id); + println!("{}", sandbox.object_id()); } return Ok(()); } if names_only { for sandbox in sandboxes { - println!("{}", sandbox.name); + println!("{}", sandbox.object_name()); } return Ok(()); } @@ -2972,23 +3471,16 @@ pub async fn sandbox_list( // Calculate column widths let name_width = sandboxes .iter() - .map(|s| s.name.len()) + .map(|s| s.object_name().len()) .max() .unwrap_or(4) .max(4); - let ns_width = sandboxes - .iter() - .map(|s| s.namespace.len()) - .max() - .unwrap_or(9) - .max(9); let created_width = 19; // "YYYY-MM-DD HH:MM:SS" // Print header println!( - "{: phase.dimmed().to_string(), _ => phase.to_string(), }; - let created = format_epoch_ms(sandbox.created_at_ms); + let created = format_epoch_ms(sandbox.metadata.as_ref().map_or(0, |m| m.created_at_ms)); println!( - "{: Result< println!("{}", "Provider:".cyan().bold()); println!(); - println!(" {} {}", "Id:".dimmed(), provider.id); - println!(" {} {}", "Name:".dimmed(), provider.name); + println!(" {} {}", "Id:".dimmed(), provider.object_id()); + println!(" {} {}", "Name:".dimmed(), provider.object_name()); println!(" {} {}", "Type:".dimmed(), provider.r#type); println!( " {} {}", @@ -3540,14 +4054,14 @@ pub async fn provider_list( if names_only { for provider in providers { - println!("{}", provider.name); + println!("{}", provider.object_name()); } return Ok(()); } let name_width = providers .iter() - .map(|provider| provider.name.len()) + .map(|provider| provider.object_name().len()) .max() .unwrap_or(4) .max(4); @@ -3569,7 +4083,7 @@ pub async fn provider_list( for provider in providers { println!( "{: Result<()> { + let mut client = grpc_client(server, tls).await?; + let response = client + .list_provider_profiles(ListProviderProfilesRequest { + limit: 100, + offset: 0, + }) + .await + .into_diagnostic()?; + let mut profiles = response.into_inner().profiles; + profiles.sort_by(|left, right| { + left.category + .cmp(&right.category) + .then_with(|| left.id.cmp(&right.id)) + }); + + if profiles.is_empty() { + println!("No provider profiles found."); + return Ok(()); + } + + println!("{}", "Available Provider Profiles:".cyan().bold()); + let mut current_category = i32::MIN; + for profile in profiles { + if profile.category != current_category { + current_category = profile.category; + println!(); + println!(" {}", display_provider_category(current_category).bold()); + } + print_provider_type_row(&profile); + } + + Ok(()) +} + +fn display_provider_category(category: i32) -> &'static str { + match ProviderProfileCategory::try_from(category).unwrap_or(ProviderProfileCategory::Other) { + ProviderProfileCategory::Inference => "INFERENCE", + ProviderProfileCategory::Agent => "AGENT", + ProviderProfileCategory::SourceControl => "SOURCE CONTROL", + ProviderProfileCategory::Messaging => "MESSAGING", + ProviderProfileCategory::Data => "DATA", + ProviderProfileCategory::Knowledge => "KNOWLEDGE", + ProviderProfileCategory::Other | ProviderProfileCategory::Unspecified => "OTHER", + } +} + +fn print_provider_type_row(profile: &ProviderProfile) { + let inference = if profile.inference_capable { + " inference" + } else { + "" + }; + println!( + " {:<12} {:<42} endpoints: {:<2}{}", + profile.id, + profile.display_name, + profile.endpoints.len(), + inference + ); +} + pub async fn provider_update( server: &str, name: &str, @@ -3632,8 +4208,12 @@ pub async fn provider_update( let response = client .update_provider(UpdateProviderRequest { provider: Some(Provider { - id: String::new(), - name: name.to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: String::new(), credentials: credential_map, config: config_map, @@ -3647,7 +4227,11 @@ pub async fn provider_update( .provider .ok_or_else(|| miette::miette!("provider missing from response"))?; - println!("{} Updated provider {}", "✓".green().bold(), provider.name); + println!( + "{} Updated provider {}", + "✓".green().bold(), + provider.object_name() + ); Ok(()) } @@ -4144,6 +4728,16 @@ fn format_setting_value(value: Option<&SettingValue>) -> String { } } +fn short_hash(hash: &str) -> &str { + if hash.len() >= 12 { &hash[..12] } else { hash } +} + +fn print_policy_merge_warnings(warnings: &[openshell_policy::PolicyMergeWarning]) { + for warning in warnings { + eprintln!("{} {}", "!".yellow().bold(), warning); + } +} + pub async fn sandbox_policy_set_global( server: &str, policy_path: &str, @@ -4172,6 +4766,7 @@ pub async fn sandbox_policy_set_global( setting_value: None, delete_setting: false, global: true, + merge_operations: vec![], }) .await .into_diagnostic()? @@ -4209,7 +4804,7 @@ pub async fn sandbox_settings_get( let response = client .get_sandbox_config(GetSandboxConfigRequest { - sandbox_id: sandbox.id.clone(), + sandbox_id: sandbox.object_id().to_string(), }) .await .into_diagnostic()? @@ -4221,16 +4816,15 @@ pub async fn sandbox_settings_get( return Ok(()); } - let policy_source = - if response.policy_source == openshell_core::proto::PolicySource::Global as i32 { - "global" - } else { - "sandbox" - }; + let policy_source = if response.policy_source == PolicySource::Global as i32 { + "global" + } else { + "sandbox" + }; - println!("Sandbox: {}", name); + println!("Sandbox: {name}"); println!("Config Rev: {}", response.config_revision); - println!("Policy Source: {}", policy_source); + println!("Policy Source: {policy_source}"); println!("Policy Hash: {}", response.policy_hash); if response.settings.is_empty() { @@ -4297,12 +4891,11 @@ fn settings_to_json_sandbox( name: &str, response: &openshell_core::proto::GetSandboxConfigResponse, ) -> serde_json::Value { - let policy_source = - if response.policy_source == openshell_core::proto::PolicySource::Global as i32 { - "global" - } else { - "sandbox" - }; + let policy_source = if response.policy_source == PolicySource::Global as i32 { + "global" + } else { + "sandbox" + }; let mut settings = serde_json::Map::new(); let mut keys: Vec<_> = response.settings.keys().cloned().collect(); @@ -4371,6 +4964,7 @@ pub async fn gateway_setting_set( setting_value: Some(setting_value), delete_setting: false, global: true, + merge_operations: vec![], }) .await .into_diagnostic()? @@ -4404,6 +4998,7 @@ pub async fn sandbox_setting_set( setting_value: Some(setting_value), delete_setting: false, global: false, + merge_operations: vec![], }) .await .into_diagnostic()? @@ -4437,6 +5032,7 @@ pub async fn gateway_setting_delete( setting_value: None, delete_setting: true, global: true, + merge_operations: vec![], }) .await .into_diagnostic()? @@ -4450,7 +5046,7 @@ pub async fn gateway_setting_delete( response.settings_revision ); } else { - println!("{} Global setting {} not found", "!".yellow(), key,); + println!("{} Global setting {} not found", "!".yellow(), key); } Ok(()) } @@ -4470,6 +5066,7 @@ pub async fn sandbox_setting_delete( setting_value: None, delete_setting: true, global: false, + merge_operations: vec![], }) .await .into_diagnostic()? @@ -4527,6 +5124,7 @@ pub async fn sandbox_policy_set( setting_value: None, delete_setting: false, global: false, + merge_operations: vec![], }) .await .into_diagnostic()?; @@ -4614,6 +5212,180 @@ pub async fn sandbox_policy_set( } } +#[allow(clippy::too_many_arguments)] +pub async fn sandbox_policy_update( + server: &str, + name: &str, + add_endpoints: &[String], + remove_endpoints: &[String], + add_deny: &[String], + add_allow: &[String], + remove_rules: &[String], + binaries: &[String], + rule_name: Option<&str>, + dry_run: bool, + wait: bool, + timeout_secs: u64, + tls: &TlsOptions, +) -> Result<()> { + if dry_run && wait { + return Err(miette!("--wait cannot be combined with --dry-run")); + } + + let plan = build_policy_update_plan( + add_endpoints, + remove_endpoints, + add_deny, + add_allow, + remove_rules, + binaries, + rule_name, + )?; + + let mut client = grpc_client(server, tls).await?; + let sandbox = client + .get_sandbox(GetSandboxRequest { + name: name.to_string(), + }) + .await + .into_diagnostic()? + .into_inner() + .sandbox + .ok_or_else(|| miette!("sandbox not found"))?; + + let sandbox_id = if sandbox.object_id().is_empty() { + return Err(miette!("sandbox missing metadata")); + } else { + sandbox.object_id().to_string() + }; + + let current = client + .get_sandbox_config(GetSandboxConfigRequest { sandbox_id }) + .await + .into_diagnostic()? + .into_inner(); + + if current.policy_source == PolicySource::Global as i32 { + return Err(miette!( + "policy is managed globally; delete the global policy before using `openshell policy update`" + )); + } + + let merged = openshell_policy::merge_policy( + current.policy.clone().unwrap_or_default(), + &plan.preview_operations, + ) + .map_err(|error| miette!("{error}"))?; + + if dry_run { + eprintln!( + "{} Dry run preview for {} incremental policy operation(s)", + "✓".green().bold(), + plan.preview_operations.len() + ); + print_policy_merge_warnings(&merged.warnings); + print_sandbox_policy(&merged.policy); + return Ok(()); + } + + let current_version = current.version; + let current_hash = current.policy_hash.clone(); + let response = client + .update_config(UpdateConfigRequest { + name: name.to_string(), + policy: None, + setting_key: String::new(), + setting_value: None, + delete_setting: false, + global: false, + merge_operations: plan.merge_operations, + }) + .await + .into_diagnostic()? + .into_inner(); + + print_policy_merge_warnings(&merged.warnings); + + if response.version == current_version && response.policy_hash == current_hash { + eprintln!( + "{} Policy unchanged (version {}, hash: {})", + "·".dimmed(), + response.version, + short_hash(&response.policy_hash) + ); + return Ok(()); + } + + eprintln!( + "{} Policy version {} submitted (hash: {})", + "✓".green().bold(), + response.version, + short_hash(&response.policy_hash) + ); + + if !wait { + return Ok(()); + } + + let deadline = Instant::now() + Duration::from_secs(timeout_secs); + loop { + if Instant::now() > deadline { + eprintln!( + "{} Timeout waiting for policy version {} to load", + "✗".red().bold(), + response.version + ); + std::process::exit(124); + } + + tokio::time::sleep(Duration::from_secs(1)).await; + + let status_resp = client + .get_sandbox_policy_status(GetSandboxPolicyStatusRequest { + name: name.to_string(), + version: response.version, + global: false, + }) + .await + .into_diagnostic()?; + + let inner = status_resp.into_inner(); + if let Some(rev) = &inner.revision { + let status = PolicyStatus::try_from(rev.status).unwrap_or(PolicyStatus::Unspecified); + match status { + PolicyStatus::Loaded => { + eprintln!( + "{} Policy version {} loaded (active version: {})", + "✓".green().bold(), + rev.version, + inner.active_version + ); + return Ok(()); + } + PolicyStatus::Failed => { + eprintln!( + "{} Policy version {} failed to load: {}", + "✗".red().bold(), + rev.version, + rev.load_error + ); + std::process::exit(1); + } + PolicyStatus::Superseded => { + eprintln!( + "{} Policy version {} was superseded (active version: {})", + "⚠".yellow().bold(), + rev.version, + inner.active_version + ); + return Ok(()); + } + _ => {} + } + } + } +} + pub async fn sandbox_policy_get( server: &str, name: &str, @@ -4797,6 +5569,7 @@ fn print_policy_revision_table(revisions: &[openshell_core::proto::SandboxPolicy // Sandbox logs command // --------------------------------------------------------------------------- +#[allow(clippy::too_many_arguments)] // user-facing CLI command pub async fn sandbox_logs( server: &str, name: &str, @@ -4845,7 +5618,7 @@ pub async fn sandbox_logs( // Streaming mode: use WatchSandbox. let mut stream = client .watch_sandbox(WatchSandboxRequest { - id: sandbox.id.clone(), + id: sandbox.object_id().to_string(), follow_status: false, follow_logs: true, follow_events: false, @@ -4872,7 +5645,7 @@ pub async fn sandbox_logs( // One-shot mode: use GetSandboxLogs. let resp = client .get_sandbox_logs(GetSandboxLogsRequest { - sandbox_id: sandbox.id.clone(), + sandbox_id: sandbox.object_id().to_string(), lines, since_ms, sources: source_filter, @@ -5196,13 +5969,14 @@ fn format_timestamp_ms(ms: i64) -> String { #[cfg(test)] mod tests { use super::{ - GatewayControlTarget, TlsOptions, format_gateway_select_header, - format_gateway_select_items, gateway_add, gateway_auth_label, gateway_select_with, - gateway_type_label, git_sync_files, http_health_check, image_requests_gpu, - inferred_provider_type, parse_cli_setting_value, parse_credential_pairs, - plaintext_gateway_is_remote, provisioning_timeout_message, ready_false_condition_message, - resolve_gateway_control_target_from, sandbox_should_persist, shell_escape, - source_requests_gpu, validate_gateway_name, validate_ssh_host, + GatewayControlTarget, TlsOptions, dockerfile_sources_supported_for_gateway, + format_gateway_select_header, format_gateway_select_items, gateway_add, gateway_auth_label, + gateway_select_with, gateway_type_label, git_sync_files, http_health_check, + image_requests_gpu, inferred_provider_type, parse_cli_setting_value, + parse_credential_pairs, plaintext_gateway_is_remote, provisioning_timeout_message, + ready_false_condition_message, resolve_from, resolve_gateway_control_target_from, + sandbox_should_persist, shell_escape, source_requests_gpu, validate_gateway_name, + validate_ssh_host, }; use crate::TEST_ENV_LOCK; use hyper::StatusCode; @@ -5273,12 +6047,9 @@ mod tests { name: name.to_string(), gateway_endpoint: endpoint.to_string(), is_remote: true, - gateway_port: 0, - remote_host: None, - resolved_host: None, auth_mode: Some("cloudflare_jwt".to_string()), - edge_team_domain: None, - edge_auth_url: None, + client_lifecycle_managed: Some(false), + ..Default::default() } } @@ -5452,6 +6223,103 @@ mod tests { assert!(!source_requests_gpu("base")); } + #[test] + fn resolve_from_classifies_existing_dockerfile_path() { + let temp = tempfile::tempdir().expect("failed to create tempdir"); + let dockerfile = temp.path().join("Dockerfile"); + fs::write(&dockerfile, "FROM scratch\n").expect("failed to write Dockerfile"); + + match resolve_from(dockerfile.to_str().expect("temp path is not UTF-8")) + .expect("expected Dockerfile source") + { + super::ResolvedSource::Dockerfile { + dockerfile: resolved, + context, + } => { + assert_eq!( + resolved, + dockerfile + .canonicalize() + .expect("failed to canonicalize Dockerfile") + ); + assert_eq!( + context, + temp.path() + .canonicalize() + .expect("failed to canonicalize context") + ); + } + super::ResolvedSource::Image(image) => { + panic!("expected Dockerfile source, got image {image}"); + } + } + } + + #[test] + fn resolve_from_rejects_missing_explicit_dockerfile_path() { + let temp = tempfile::tempdir().expect("failed to create tempdir"); + let missing = temp.path().join("Dockerfile"); + + let err = resolve_from(missing.to_str().expect("temp path is not UTF-8")) + .expect_err("expected missing Dockerfile path to be rejected"); + + assert!( + err.to_string().contains("local --from path does not exist"), + "unexpected error: {err}" + ); + } + + #[test] + fn resolve_from_keeps_dockerfile_named_image_refs_as_images() { + let image_ref = "ghcr.io/acme/dockerfile-runner:latest"; + + match resolve_from(image_ref).expect("expected image source") { + super::ResolvedSource::Image(image) => assert_eq!(image, image_ref), + super::ResolvedSource::Dockerfile { .. } => { + panic!("expected image ref, got Dockerfile source"); + } + } + } + + #[test] + fn dockerfile_sources_are_rejected_for_remote_gateways() { + let metadata = GatewayMetadata { + name: "remote".to_string(), + gateway_endpoint: "https://gateway.example.com".to_string(), + is_remote: true, + gateway_port: 443, + remote_host: Some("user@gateway.example.com".to_string()), + resolved_host: Some("gateway.example.com".to_string()), + auth_mode: None, + edge_team_domain: None, + edge_auth_url: None, + vm_driver_state_dir: None, + ..Default::default() + }; + + assert!(!dockerfile_sources_supported_for_gateway(Some(&metadata))); + } + + #[test] + fn dockerfile_sources_are_allowed_for_local_gateways() { + let metadata = GatewayMetadata { + name: "local".to_string(), + gateway_endpoint: "http://127.0.0.1:8080".to_string(), + is_remote: false, + gateway_port: 8080, + remote_host: None, + resolved_host: None, + auth_mode: None, + edge_team_domain: None, + edge_auth_url: None, + vm_driver_state_dir: None, + ..Default::default() + }; + + assert!(dockerfile_sources_supported_for_gateway(Some(&metadata))); + assert!(dockerfile_sources_supported_for_gateway(None)); + } + #[test] fn ready_false_condition_message_prefers_reason_and_message() { let status = SandboxStatus { @@ -5581,6 +6449,49 @@ mod tests { } } + #[test] + fn resolve_gateway_control_target_non_managed_loopback_is_external() { + // A gateway registered via `gateway add http://localhost:8080` should + // be classified as an external registration, not a local container. + let metadata = GatewayMetadata { + name: "localhost".to_string(), + gateway_endpoint: "http://localhost:8080".to_string(), + auth_mode: Some("plaintext".to_string()), + client_lifecycle_managed: Some(false), + ..Default::default() + }; + let target = resolve_gateway_control_target_from(Some(metadata), None); + assert!(matches!(target, GatewayControlTarget::ExternalRegistration)); + } + + #[test] + fn resolve_gateway_control_target_managed_gateway_is_local() { + // A gateway deployed via `gateway start` should be classified as local. + let metadata = GatewayMetadata { + name: "openshell".to_string(), + gateway_endpoint: "https://127.0.0.1:8080".to_string(), + gateway_port: 8080, + client_lifecycle_managed: Some(true), + ..Default::default() + }; + let target = resolve_gateway_control_target_from(Some(metadata), None); + assert!(matches!(target, GatewayControlTarget::Local)); + } + + #[test] + fn resolve_gateway_control_target_legacy_metadata_defaults_to_local() { + // Legacy metadata without the `client_lifecycle_managed` field + // should preserve the existing behavior: non-remote → Local. + let metadata = GatewayMetadata { + name: "openshell".to_string(), + gateway_endpoint: "https://127.0.0.1:8080".to_string(), + gateway_port: 8080, + ..Default::default() + }; + let target = resolve_gateway_control_target_from(Some(metadata), None); + assert!(matches!(target, GatewayControlTarget::Local)); + } + #[test] fn gateway_select_uses_explicit_name_without_prompting() { let tmpdir = tempfile::tempdir().expect("create tmpdir"); @@ -5660,13 +6571,8 @@ mod tests { GatewayMetadata { name: "local".to_string(), gateway_endpoint: "http://127.0.0.1:8080".to_string(), - is_remote: false, gateway_port: 8080, - remote_host: None, - resolved_host: None, - auth_mode: None, - edge_team_domain: None, - edge_auth_url: None, + ..Default::default() }, ]; @@ -5695,13 +6601,8 @@ mod tests { let gateway = GatewayMetadata { name: "local".to_string(), gateway_endpoint: "https://127.0.0.1:8080".to_string(), - is_remote: false, gateway_port: 8080, - remote_host: None, - resolved_host: None, - auth_mode: None, - edge_team_domain: None, - edge_auth_url: None, + ..Default::default() }; assert_eq!(gateway_auth_label(&gateway), "mtls"); @@ -5742,25 +6643,40 @@ mod tests { #[test] fn gateway_add_registers_plaintext_loopback_gateway_without_local_flag() { + let _ = rustls::crypto::ring::default_provider().install_default(); let tmpdir = tempfile::tempdir().expect("create tmpdir"); with_tmp_xdg(tmpdir.path(), || { let runtime = tokio::runtime::Runtime::new().expect("create runtime"); runtime.block_on(async { - gateway_add("http://127.0.0.1:8080", None, None, None, false) - .await - .expect("register plaintext gateway"); + gateway_add( + "http://127.0.0.1:8080", + None, + None, + None, + false, + None, + "openshell-cli", + None, + None, + ) + .await + .expect("register plaintext gateway"); }); - let metadata = load_gateway_metadata("127.0.0.1").expect("load stored gateway"); + // Loopback endpoints derive the canonical "openshell" gateway + // name, matching init-pki.sh and default_tls_dir conventions. + let metadata = load_gateway_metadata("openshell").expect("load stored gateway"); assert_eq!(metadata.auth_mode.as_deref(), Some("plaintext")); assert!(!metadata.is_remote); + assert_eq!(metadata.client_lifecycle_managed, Some(false)); assert_eq!(metadata.gateway_endpoint, "http://127.0.0.1:8080"); - assert_eq!(load_active_gateway().as_deref(), Some("127.0.0.1")); + assert_eq!(load_active_gateway().as_deref(), Some("openshell")); }); } #[test] fn gateway_add_respects_local_flag_for_plaintext_registrations() { + let _ = rustls::crypto::ring::default_provider().install_default(); let tmpdir = tempfile::tempdir().expect("create tmpdir"); with_tmp_xdg(tmpdir.path(), || { let runtime = tokio::runtime::Runtime::new().expect("create runtime"); @@ -5771,6 +6687,10 @@ mod tests { None, None, true, + None, + "openshell-cli", + None, + None, ) .await .expect("register plaintext gateway"); @@ -5779,6 +6699,7 @@ mod tests { let metadata = load_gateway_metadata("dev-http").expect("load stored gateway"); assert_eq!(metadata.auth_mode.as_deref(), Some("plaintext")); assert!(!metadata.is_remote); + assert_eq!(metadata.client_lifecycle_managed, Some(false)); assert_eq!(metadata.gateway_endpoint, "http://gateway.example.com:8080"); assert_eq!(load_active_gateway().as_deref(), Some("dev-http")); }); @@ -5786,6 +6707,7 @@ mod tests { #[tokio::test] async fn http_health_check_supports_plain_http_endpoints() { + let _ = rustls::crypto::ring::default_provider().install_default(); let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); let addr = listener.local_addr().expect("listener addr"); let server = thread::spawn(move || { diff --git a/crates/openshell-cli/src/ssh.rs b/crates/openshell-cli/src/ssh.rs index ebcbbeb4f6..89e9071e16 100644 --- a/crates/openshell-cli/src/ssh.rs +++ b/crates/openshell-cli/src/ssh.rs @@ -7,14 +7,16 @@ use crate::tls::{TlsOptions, build_rustls_config, grpc_client, require_tls_mater use miette::{IntoDiagnostic, Result, WrapErr}; #[cfg(unix)] use nix::sys::signal::{SaFlags, SigAction, SigHandler, SigSet, Signal, sigaction}; +use openshell_core::ObjectId; use openshell_core::forward::{ - find_ssh_forward_pid, resolve_ssh_gateway, shell_escape, write_forward_pid, + build_proxy_command, find_ssh_forward_pid, resolve_ssh_gateway, shell_escape, + validate_ssh_session_response, write_forward_pid, }; use openshell_core::proto::{CreateSshSessionRequest, GetSandboxRequest}; use owo_colors::OwoColorize; use rustls::pki_types::ServerName; use std::fs; -use std::io::IsTerminal; +use std::io::{IsTerminal, Write}; #[cfg(unix)] use std::os::unix::process::CommandExt; use std::path::{Path, PathBuf}; @@ -42,7 +44,7 @@ impl Editor { } } - fn remote_target(self, host_alias: &str) -> String { + fn remote_target(host_alias: &str) -> String { format!("ssh-remote+{host_alias}") } @@ -81,16 +83,18 @@ async fn ssh_session_config( let response = client .create_ssh_session(CreateSshSessionRequest { - sandbox_id: sandbox.id, + sandbox_id: sandbox.object_id().to_string(), }) .await .into_diagnostic()?; let session = response.into_inner(); + validate_ssh_session_response(&session) + .map_err(|err| miette::miette!("gateway returned invalid SSH session response: {err}"))?; let exe = std::env::current_exe() .into_diagnostic() .wrap_err("failed to resolve OpenShell executable")?; - let exe_command = shell_escape(&exe.to_string_lossy()); + let exe_command = exe.to_string_lossy().into_owned(); // When using Cloudflare bearer auth, the SSH CONNECT must go through the // external tunnel endpoint (the cluster URL), not the server's internal @@ -114,12 +118,12 @@ async fn ssh_session_config( let gateway_name = tls .gateway_name() .ok_or_else(|| miette::miette!("gateway name is required to build SSH proxy command"))?; - let proxy_command = format!( - "{exe_command} ssh-proxy --gateway {} --sandbox-id {} --token {} --gateway-name {}", - gateway_url, - session.sandbox_id, - session.token, - shell_escape(gateway_name), + let proxy_command = build_proxy_command( + &exe_command, + &gateway_url, + &session.sandbox_id, + &session.token, + gateway_name, ); Ok(SshSessionConfig { @@ -131,6 +135,11 @@ async fn ssh_session_config( } fn ssh_base_command(proxy_command: &str) -> Command { + // SSH log level follows the program's verbosity. main() maps the `-v` + // count to OPENSHELL_SSH_LOG_LEVEL; an explicit env-var override wins. + let ssh_log_level = + std::env::var("OPENSHELL_SSH_LOG_LEVEL").unwrap_or_else(|_| "ERROR".to_string()); + let mut command = Command::new("ssh"); command .arg("-o") @@ -142,7 +151,15 @@ fn ssh_base_command(proxy_command: &str) -> Command { .arg("-o") .arg("GlobalKnownHostsFile=/dev/null") .arg("-o") - .arg("LogLevel=ERROR"); + .arg(format!("LogLevel={ssh_log_level}")) + // Detect a dead relay within ~45s. The relay rides on a TCP connection + // that the client has no way to observe silently dropping (gateway + // restart, supervisor restart, cluster failover), so fall back to + // SSH-level keepalives instead of hanging forever. + .arg("-o") + .arg("ServerAliveInterval=15") + .arg("-o") + .arg("ServerAliveCountMax=3"); command } @@ -340,12 +357,13 @@ pub async fn sandbox_forward( command.status().await.into_diagnostic()? } else { let mut child = command.spawn().into_diagnostic()?; - match tokio::time::timeout(FOREGROUND_FORWARD_STARTUP_GRACE_PERIOD, child.wait()).await { - Ok(status) => status.into_diagnostic()?, - Err(_) => { - eprintln!("{}", foreground_forward_started_message(name, spec)); - child.wait().await.into_diagnostic()? - } + if let Ok(status) = + tokio::time::timeout(FOREGROUND_FORWARD_STARTUP_GRACE_PERIOD, child.wait()).await + { + status.into_diagnostic()? + } else { + eprintln!("{}", foreground_forward_started_message(name, spec)); + child.wait().await.into_diagnostic()? } }; @@ -459,9 +477,67 @@ enum UploadSource { FileList { base_dir: PathBuf, files: Vec, + archive_prefix: Option, }, } +fn write_upload_archive(writer: W, source: UploadSource) -> Result<()> { + let mut archive = tar::Builder::new(writer); + match source { + UploadSource::SinglePath { + local_path, + tar_name, + } => { + if local_path.is_file() { + archive + .append_path_with_name(&local_path, &tar_name) + .into_diagnostic()?; + } else if local_path.is_dir() { + archive + .append_dir_all(&tar_name, &local_path) + .into_diagnostic()?; + } else { + return Err(miette::miette!( + "local path does not exist: {}", + local_path.display() + )); + } + } + UploadSource::FileList { + base_dir, + files, + archive_prefix, + } => { + for file in &files { + let full_path = base_dir.join(file); + let archive_path = archive_prefix + .as_ref() + .map_or_else(|| PathBuf::from(file), |prefix| prefix.join(file)); + if full_path.is_file() { + archive + .append_path_with_name(&full_path, &archive_path) + .into_diagnostic() + .wrap_err_with(|| { + format!("failed to add {} to tar archive", archive_path.display()) + })?; + } else if full_path.is_dir() { + archive + .append_dir_all(&archive_path, &full_path) + .into_diagnostic() + .wrap_err_with(|| { + format!( + "failed to add directory {} to tar archive", + archive_path.display() + ) + })?; + } + } + } + } + archive.finish().into_diagnostic()?; + Ok(()) +} + /// Core tar-over-SSH upload: streams a tar archive into `dest_dir` on the /// sandbox. Callers are responsible for splitting the destination path so /// that `dest_dir` is always a directory. @@ -480,10 +556,7 @@ async fn ssh_tar_upload( // When no explicit destination is given, use the unescaped `$HOME` shell // variable so the remote shell resolves it at runtime. - let escaped_dest = match dest_dir { - Some(d) => shell_escape(d), - None => "$HOME".to_string(), - }; + let escaped_dest = dest_dir.map_or_else(|| "$HOME".to_string(), shell_escape); let mut ssh = ssh_base_command(&session.proxy_command); ssh.arg("-T") @@ -504,50 +577,9 @@ async fn ssh_tar_upload( .ok_or_else(|| miette::miette!("failed to open stdin for ssh process"))?; // Build the tar archive in a blocking task since the tar crate is synchronous. - tokio::task::spawn_blocking(move || -> Result<()> { - let mut archive = tar::Builder::new(stdin); - match source { - UploadSource::SinglePath { - local_path, - tar_name, - } => { - if local_path.is_file() { - archive - .append_path_with_name(&local_path, &tar_name) - .into_diagnostic()?; - } else if local_path.is_dir() { - archive.append_dir_all(".", &local_path).into_diagnostic()?; - } else { - return Err(miette::miette!( - "local path does not exist: {}", - local_path.display() - )); - } - } - UploadSource::FileList { base_dir, files } => { - for file in &files { - let full_path = base_dir.join(file); - if full_path.is_file() { - archive - .append_path_with_name(&full_path, file) - .into_diagnostic() - .wrap_err_with(|| format!("failed to add {file} to tar archive"))?; - } else if full_path.is_dir() { - archive - .append_dir_all(file, &full_path) - .into_diagnostic() - .wrap_err_with(|| { - format!("failed to add directory {file} to tar archive") - })?; - } - } - } - } - archive.finish().into_diagnostic()?; - Ok(()) - }) - .await - .into_diagnostic()??; + tokio::task::spawn_blocking(move || -> Result<()> { write_upload_archive(stdin, source) }) + .await + .into_diagnostic()??; let status = tokio::task::spawn_blocking(move || child.wait()) .await @@ -563,7 +595,7 @@ async fn ssh_tar_upload( Ok(()) } -/// Split a sandbox path into (parent_directory, basename). +/// Split a sandbox path into (`parent_directory`, basename). /// /// Examples: /// `"/sandbox/.bashrc"` -> `("/sandbox", ".bashrc")` @@ -587,6 +619,7 @@ pub async fn sandbox_sync_up_files( name: &str, base_dir: &Path, files: &[String], + local_path: &Path, dest: Option<&str>, tls: &TlsOptions, ) -> Result<()> { @@ -600,6 +633,7 @@ pub async fn sandbox_sync_up_files( UploadSource::FileList { base_dir: base_dir.to_path_buf(), files: files.to_vec(), + archive_prefix: file_list_archive_prefix(local_path), }, tls, ) @@ -629,22 +663,23 @@ pub async fn sandbox_sync_up( // passed "/sandbox"), fall through to directory semantics instead. The // sandbox user cannot write to "/" and the intent is almost certainly // "put the file inside /sandbox", not "create a file named sandbox in /". - if let Some(path) = sandbox_path { - if local_path.is_file() && !path.ends_with('/') { - let (parent, target_name) = split_sandbox_path(path); - if parent != "/" { - return ssh_tar_upload( - server, - name, - Some(parent), - UploadSource::SinglePath { - local_path: local_path.to_path_buf(), - tar_name: target_name.into(), - }, - tls, - ) - .await; - } + if let Some(path) = sandbox_path + && local_path.is_file() + && !path.ends_with('/') + { + let (parent, target_name) = split_sandbox_path(path); + if parent != "/" { + return ssh_tar_upload( + server, + name, + Some(parent), + UploadSource::SinglePath { + local_path: local_path.to_path_buf(), + tar_name: target_name.into(), + }, + tls, + ) + .await; } } @@ -654,8 +689,11 @@ pub async fn sandbox_sync_up( .ok_or_else(|| miette::miette!("path has no file name"))? .to_os_string() } else { - // For directories the tar_name is unused — append_dir_all uses "." - ".".into() + // For directories, wrap contents under the source basename so uploads + // land at `//...` — matches `scp -r` and `cp -r`. Falls + // back to "." for paths with no meaningful basename (`.`, `/`), which + // preserves the legacy flatten behavior in those edge cases. + directory_upload_prefix(local_path) }; ssh_tar_upload( @@ -671,6 +709,31 @@ pub async fn sandbox_sync_up( .await } +/// Compute the tar entry prefix for a directory upload. +/// +/// Returns the directory's basename for any path with a meaningful basename; +/// callers extracting at `` will see contents wrapped under +/// `//...`. Returns `"."` for paths without a basename +/// (e.g. `.` or `/`), which produces flat extraction at ``. +fn directory_upload_prefix(local_path: &Path) -> std::ffi::OsString { + local_path + .file_name() + .map_or_else(|| ".".into(), std::ffi::OsStr::to_os_string) +} + +fn file_list_archive_prefix(local_path: &Path) -> Option { + if !local_path.is_dir() { + return None; + } + + let prefix = directory_upload_prefix(local_path); + if prefix == "." { + None + } else { + Some(PathBuf::from(prefix)) + } +} + /// Pull a path from a sandbox to a local destination using tar-over-SSH. pub async fn sandbox_sync_down( server: &str, @@ -758,6 +821,13 @@ pub async fn sandbox_ssh_proxy( token: &str, tls: &TlsOptions, ) -> Result<()> { + // The gateway returns 412 (Precondition Failed) when the sandbox pod + // exists but hasn't reached Ready phase yet. This is a transient state + // after sandbox allocation — retry with backoff instead of failing + // immediately. + const MAX_CONNECT_WAIT: Duration = Duration::from_secs(60); + const INITIAL_BACKOFF: Duration = Duration::from_secs(1); + let url: url::Url = gateway_url .parse() .into_diagnostic() @@ -776,13 +846,6 @@ pub async fn sandbox_ssh_proxy( "CONNECT {connect_path} HTTP/1.1\r\nHost: {gateway_host}\r\nX-Sandbox-Id: {sandbox_id}\r\nX-Sandbox-Token: {token}\r\n\r\n" ); - // The gateway returns 412 (Precondition Failed) when the sandbox pod - // exists but hasn't reached Ready phase yet. This is a transient state - // after sandbox allocation — retry with backoff instead of failing - // immediately. - const MAX_CONNECT_WAIT: Duration = Duration::from_secs(60); - const INITIAL_BACKOFF: Duration = Duration::from_secs(1); - let start = std::time::Instant::now(); let mut backoff = INITIAL_BACKOFF; let mut buf_stream; @@ -867,10 +930,14 @@ fn render_ssh_config(gateway: &str, name: &str) -> String { let exe = std::env::current_exe().expect("failed to resolve OpenShell executable"); let exe = shell_escape(&exe.to_string_lossy()); - let proxy_cmd = format!("{exe} ssh-proxy --gateway-name {gateway} --name {name}"); + let proxy_cmd = format!( + "{exe} ssh-proxy --gateway-name {} --name {}", + shell_escape(gateway), + shell_escape(name), + ); let host_alias = host_alias(name); format!( - "Host {host_alias}\n User sandbox\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n GlobalKnownHostsFile /dev/null\n LogLevel ERROR\n ProxyCommand {proxy_cmd}\n" + "Host {host_alias}\n User sandbox\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n GlobalKnownHostsFile /dev/null\n LogLevel ERROR\n ServerAliveInterval 15\n ServerAliveCountMax 3\n ProxyCommand {proxy_cmd}\n" ) } @@ -968,8 +1035,7 @@ fn upsert_host_block(contents: &str, alias: &str, block: &str) -> String { .enumerate() .skip(start + 1) .find(|(_, line)| line.trim_start().starts_with("Host ")) - .map(|(idx, _)| idx) - .unwrap_or(lines.len()); + .map_or(lines.len(), |(idx, _)| idx); out.extend_from_slice(&lines[..start]); if !out.is_empty() && !out.last().is_some_and(|line| line.is_empty()) { @@ -1018,7 +1084,7 @@ fn launch_editor(editor: Editor, host_alias: &str) -> Result<()> { launch_editor_command( editor.binary(), editor.label(), - &editor.remote_target(host_alias), + &Editor::remote_target(host_alias), ) } @@ -1075,15 +1141,11 @@ async fn connect_gateway( port: u16, tls: &TlsOptions, ) -> Result> { - // When using edge bearer auth, route through the WebSocket tunnel proxy - // regardless of the origin scheme. The proxy handles edge auth headers - // and TLS termination at the edge; the origin may be plaintext HTTP - // behind the tunnel. - if tls.is_bearer_auth() { - let token = tls - .edge_token - .as_deref() - .ok_or_else(|| miette::miette!("edge token required for tunnel"))?; + // When using Cloudflare edge bearer auth, route through the WebSocket + // tunnel proxy regardless of the origin scheme. The proxy handles edge + // auth headers and TLS termination at the edge; the origin may be + // plaintext HTTP behind the tunnel. OIDC tokens bypass the tunnel. + if let Some(token) = tls.edge_token.as_deref() { let gateway_url = format!("https://{host}:{port}"); let proxy = crate::edge_tunnel::start_tunnel_proxy(&gateway_url, token).await?; let tcp = TcpStream::connect(proxy.local_addr) @@ -1174,6 +1236,7 @@ mod tests { } #[test] + #[allow(unsafe_code)] // Test-only: env vars require unsafe in Rust 2024. fn install_ssh_config_adds_include_once_and_updates_managed_file() { let _guard = TEST_ENV_LOCK .lock() @@ -1265,6 +1328,121 @@ mod tests { assert_eq!(split_sandbox_path("/a/b/c/d.txt"), ("/a/b/c", "d.txt")); } + #[test] + fn directory_upload_prefix_uses_basename_for_named_directories() { + assert_eq!( + directory_upload_prefix(Path::new("/tmp/upload-test")), + std::ffi::OsString::from("upload-test") + ); + assert_eq!( + directory_upload_prefix(Path::new("foo")), + std::ffi::OsString::from("foo") + ); + assert_eq!( + directory_upload_prefix(Path::new("./parent/nested")), + std::ffi::OsString::from("nested") + ); + } + + #[test] + fn directory_upload_prefix_falls_back_to_dot_for_basename_less_paths() { + assert_eq!( + directory_upload_prefix(Path::new(".")), + std::ffi::OsString::from(".") + ); + assert_eq!( + directory_upload_prefix(Path::new("/")), + std::ffi::OsString::from(".") + ); + } + + #[test] + fn file_list_archive_prefix_uses_named_directory_basename() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + let source = tmpdir.path().join("source-dir"); + let file = tmpdir.path().join("file.txt"); + fs::create_dir_all(&source).expect("create source dir"); + fs::write(&file, "file").expect("write file"); + + assert_eq!( + file_list_archive_prefix(&source), + Some(PathBuf::from("source-dir")) + ); + assert_eq!(file_list_archive_prefix(Path::new(".")), None); + assert_eq!(file_list_archive_prefix(&file), None); + } + + fn upload_archive_paths(source: UploadSource) -> Vec { + let mut bytes = Vec::new(); + write_upload_archive(&mut bytes, source).expect("write upload archive"); + let mut archive = tar::Archive::new(std::io::Cursor::new(bytes)); + let entries = archive.entries().expect("read archive entries"); + let mut paths = entries + .map(|entry| { + entry + .expect("read archive entry") + .path() + .expect("read archive path") + .to_string_lossy() + .into_owned() + }) + .collect::>(); + paths.sort(); + paths + } + + #[test] + fn file_list_archive_preserves_directory_prefix_when_requested() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + let base_dir = tmpdir.path().join("nested"); + fs::create_dir_all(base_dir.join("inner")).expect("create dirs"); + fs::write(base_dir.join("file.txt"), "file").expect("write file"); + fs::write(base_dir.join("inner/child.txt"), "child").expect("write child"); + + let paths = upload_archive_paths(UploadSource::FileList { + base_dir, + files: vec!["file.txt".into(), "inner/child.txt".into()], + archive_prefix: Some(PathBuf::from("nested")), + }); + + assert_eq!(paths, vec!["nested/file.txt", "nested/inner/child.txt"]); + } + + #[test] + fn file_list_archive_stays_flat_without_directory_prefix() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + let base_dir = tmpdir.path().join("nested"); + fs::create_dir_all(base_dir.join("inner")).expect("create dirs"); + fs::write(base_dir.join("file.txt"), "file").expect("write file"); + fs::write(base_dir.join("inner/child.txt"), "child").expect("write child"); + + let paths = upload_archive_paths(UploadSource::FileList { + base_dir, + files: vec!["file.txt".into(), "inner/child.txt".into()], + archive_prefix: None, + }); + + assert_eq!(paths, vec!["file.txt", "inner/child.txt"]); + } + + #[test] + fn single_directory_archive_preserves_directory_basename() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + let source = tmpdir.path().join("source-dir"); + fs::create_dir_all(source.join("inner")).expect("create dirs"); + fs::write(source.join("file.txt"), "file").expect("write file"); + fs::write(source.join("inner/child.txt"), "child").expect("write child"); + + let paths = upload_archive_paths(UploadSource::SinglePath { + local_path: source, + tar_name: "source-dir".into(), + }); + + assert!(paths.contains(&"source-dir/file.txt".to_string())); + assert!(paths.contains(&"source-dir/inner/child.txt".to_string())); + assert!(paths.iter().all(|path| path.starts_with("source-dir/"))); + } + #[test] fn split_sandbox_path_handles_root_and_bare_names() { // File directly under root diff --git a/crates/openshell-cli/src/tls.rs b/crates/openshell-cli/src/tls.rs index cd64835300..c733d3db30 100644 --- a/crates/openshell-cli/src/tls.rs +++ b/crates/openshell-cli/src/tls.rs @@ -6,9 +6,11 @@ use openshell_core::proto::inference_client::InferenceClient; use openshell_core::proto::open_shell_client::OpenShellClient; use rustls::{ RootCertStore, - pki_types::{CertificateDer, PrivateKeyDer}, + client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier}, + pki_types::{CertificateDer, PrivateKeyDer, ServerName, UnixTime}, }; use std::collections::HashMap; +use std::future::Future; use std::io::Cursor; use std::net::SocketAddr; use std::path::PathBuf; @@ -34,6 +36,11 @@ pub struct TlsOptions { /// Edge auth bearer token — when set, disables mTLS client certs and /// injects authentication headers on every gRPC request instead. pub edge_token: Option, + /// OIDC bearer token — when set, injects `authorization: Bearer ` + /// on every gRPC request. Takes precedence over `edge_token`. + pub oidc_token: Option, + /// Skip TLS certificate verification for gateway connections. + pub gateway_insecure: bool, } impl TlsOptions { @@ -44,6 +51,8 @@ impl TlsOptions { key, gateway_name: None, edge_token: None, + oidc_token: None, + gateway_insecure: false, } } @@ -90,9 +99,9 @@ impl TlsOptions { } } - /// Returns `true` when using edge token auth (no mTLS client certs). + /// Returns `true` when using bearer token auth (edge or OIDC). pub fn is_bearer_auth(&self) -> bool { - self.edge_token.is_some() + self.edge_token.is_some() || self.oidc_token.is_some() } } @@ -220,6 +229,86 @@ pub fn build_tonic_tls_config(materials: &TlsMaterials) -> ClientTlsConfig { .identity(identity) } +#[derive(Debug)] +struct InsecureServerCertVerifier; + +impl ServerCertVerifier for InsecureServerCertVerifier { + fn verify_server_cert( + &self, + _end_entity: &CertificateDer<'_>, + _intermediates: &[CertificateDer<'_>], + _server_name: &ServerName<'_>, + _ocsp_response: &[u8], + _now: UnixTime, + ) -> Result { + Ok(ServerCertVerified::assertion()) + } + + fn verify_tls12_signature( + &self, + _message: &[u8], + _cert: &CertificateDer<'_>, + _dss: &rustls::DigitallySignedStruct, + ) -> Result { + Ok(HandshakeSignatureValid::assertion()) + } + + fn verify_tls13_signature( + &self, + _message: &[u8], + _cert: &CertificateDer<'_>, + _dss: &rustls::DigitallySignedStruct, + ) -> Result { + Ok(HandshakeSignatureValid::assertion()) + } + + fn supported_verify_schemes(&self) -> Vec { + rustls::crypto::ring::default_provider() + .signature_verification_algorithms + .supported_schemes() + } +} + +#[derive(Clone)] +struct InsecureTlsConnector { + tls_connector: tokio_rustls::TlsConnector, +} + +impl tower::Service for InsecureTlsConnector { + type Response = hyper_util::rt::TokioIo>; + type Error = Box; + type Future = + std::pin::Pin> + Send>>; + + fn poll_ready( + &mut self, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + std::task::Poll::Ready(Ok(())) + } + + fn call(&mut self, uri: hyper::Uri) -> Self::Future { + let tls_connector = self.tls_connector.clone(); + Box::pin(async move { + let host = uri.host().unwrap_or("localhost").to_string(); + let port = uri.port_u16().unwrap_or(443); + let addr = format!("{host}:{port}"); + let tcp = tokio::net::TcpStream::connect(addr).await?; + let server_name = ServerName::try_from(host)?; + let tls_stream = tls_connector.connect(server_name, tcp).await?; + Ok(hyper_util::rt::TokioIo::new(tls_stream)) + }) + } +} + +pub fn build_insecure_rustls_config() -> Result { + let config = rustls::ClientConfig::builder() + .dangerous() + .with_custom_certificate_verifier(std::sync::Arc::new(InsecureServerCertVerifier)) + .with_no_client_auth(); + Ok(config) +} + /// Tunnel proxy addresses keyed by upstream endpoint + token. /// /// Each distinct edge-authenticated gateway gets its own local proxy instead of @@ -258,9 +347,10 @@ pub async fn build_channel(server: &str, tls: &TlsOptions) -> Result { return endpoint.connect().await.into_diagnostic(); } - // When edge bearer auth is active and the server is HTTPS, + // When Cloudflare edge bearer auth is active and the server is HTTPS, // route traffic through a local WebSocket tunnel proxy instead. - if tls.is_bearer_auth() && server.starts_with("https://") { + // OIDC tokens bypass the tunnel — they connect directly. + if tls.edge_token.is_some() && server.starts_with("https://") { let token = tls .edge_token .as_deref() @@ -277,16 +367,51 @@ pub async fn build_channel(server: &str, tls: &TlsOptions) -> Result { return endpoint.connect().await.into_diagnostic(); } + if tls.gateway_insecure && server.starts_with("https://") { + tracing::warn!("TLS certificate verification is disabled — do not use in production"); + let rustls_config = build_insecure_rustls_config()?; + let tls_connector = tokio_rustls::TlsConnector::from(std::sync::Arc::new(rustls_config)); + let connector = InsecureTlsConnector { tls_connector }; + // Use http:// so tonic does not layer its own TLS on top — our + // connector performs TLS with the insecure config. + let http_uri = server.replacen("https://", "http://", 1); + let endpoint = Endpoint::from_shared(http_uri) + .into_diagnostic()? + .connect_timeout(Duration::from_secs(10)) + .http2_keep_alive_interval(Duration::from_secs(10)) + .keep_alive_while_idle(true); + return endpoint + .connect_with_connector(connector) + .await + .into_diagnostic(); + } + let mut endpoint = Endpoint::from_shared(server.to_string()) .into_diagnostic()? .connect_timeout(Duration::from_secs(10)) .http2_keep_alive_interval(Duration::from_secs(10)) .keep_alive_while_idle(true); - let tls_config = if tls.is_bearer_auth() { - // Bearer mode without HTTPS (e.g. http:// direct) — no tunnel needed, - // but also no TLS config to set. This branch shouldn't normally happen - // (edge endpoints are always HTTPS) but handle gracefully. + let tls_config = if tls.oidc_token.is_some() { + // OIDC bearer auth over HTTPS: use mTLS certs for the transport layer + // when available (server may still require client certs), and layer + // the Bearer token on top via the interceptor. + require_tls_materials(server, tls).map_or_else( + |_| { + let resolved = tls.with_default_paths(server); + resolved + .ca + .as_ref() + .and_then(|ca_path| std::fs::read(ca_path).ok()) + .map_or_else(ClientTlsConfig::new, |ca_pem| { + ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem)) + }) + }, + |materials| build_tonic_tls_config(&materials), + ) + } else if tls.edge_token.is_some() { + // Edge bearer mode — routed through tunnel above; if we reach here + // the server is not HTTPS so connect plaintext. return endpoint.connect().await.into_diagnostic(); } else { // Standard mTLS: private CA + client cert. @@ -308,22 +433,39 @@ pub async fn grpc_client(server: &str, tls: &TlsOptions) -> Result { Ok(OpenShellClient::with_interceptor(channel, interceptor)) } -/// Interceptor that injects edge authentication headers into every outgoing -/// gRPC request. When no token is set, acts as a no-op. +/// Interceptor that injects authentication headers into every outgoing gRPC request. /// -/// Currently sends Cloudflare Access headers for compatibility: -/// - `Cf-Access-Jwt-Assertion` header -/// - `CF_Authorization` cookie +/// Supports OIDC Bearer tokens (standard `authorization` header) and +/// Cloudflare Access tokens (custom headers). When no token is set, acts +/// as a no-op. OIDC takes precedence over edge tokens. #[derive(Clone)] +#[allow(clippy::struct_field_names)] pub struct EdgeAuthInterceptor { + /// Standard `authorization: Bearer ` for OIDC. + bearer_value: Option>, + /// CF-specific `Cf-Access-Jwt-Assertion` header. header_value: Option>, + /// CF-specific `Cookie: CF_Authorization=` header. cookie_value: Option>, } impl EdgeAuthInterceptor { /// Create an interceptor from [`TlsOptions`]. Returns a no-op interceptor - /// when no edge token is configured. + /// when no auth token is configured. pub fn maybe_from(tls: &TlsOptions) -> Result { + // OIDC bearer token takes precedence. + if let Some(ref token) = tls.oidc_token { + let bearer: tonic::metadata::MetadataValue = + format!("Bearer {token}") + .parse() + .map_err(|_| miette::miette!("invalid OIDC token value"))?; + return Ok(Self { + bearer_value: Some(bearer), + header_value: None, + cookie_value: None, + }); + } + let (header_value, cookie_value) = match tls.edge_token.as_deref() { Some(t) => { let hv: tonic::metadata::MetadataValue = t @@ -338,6 +480,7 @@ impl EdgeAuthInterceptor { None => (None, None), }; Ok(Self { + bearer_value: None, header_value, cookie_value, }) @@ -349,6 +492,9 @@ impl tonic::service::Interceptor for EdgeAuthInterceptor { &mut self, mut req: tonic::Request<()>, ) -> std::result::Result, tonic::Status> { + if let Some(ref val) = self.bearer_value { + req.metadata_mut().insert("authorization", val.clone()); + } if let Some(ref val) = self.header_value { req.metadata_mut() .insert("cf-access-jwt-assertion", val.clone()); diff --git a/crates/openshell-cli/tests/ensure_providers_integration.rs b/crates/openshell-cli/tests/ensure_providers_integration.rs index 2cd3620239..0b29f73f46 100644 --- a/crates/openshell-cli/tests/ensure_providers_integration.rs +++ b/crates/openshell-cli/tests/ensure_providers_integration.rs @@ -11,14 +11,16 @@ use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer}; use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - ExecSandboxEvent, ExecSandboxRequest, GetGatewayConfigRequest, GetGatewayConfigResponse, - GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, - GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxesRequest, ListSandboxesResponse, Provider, ProviderResponse, - RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, - ServiceStatus, UpdateProviderRequest, WatchSandboxRequest, + ExecSandboxEvent, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, + GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, + Provider, ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, + SandboxStreamEvent, ServiceStatus, SupervisorMessage, UpdateProviderRequest, + WatchSandboxRequest, }; +use openshell_core::{ObjectId, ObjectName}; use rcgen::{ BasicConstraints, Certificate, CertificateParams, ExtendedKeyUsagePurpose, IsCa, KeyPair, }; @@ -104,8 +106,12 @@ impl TestOpenShell { providers.insert( name.to_string(), Provider { - id: format!("id-{name}"), - name: name.to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("id-{name}"), + name: name.to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: provider_type.to_string(), credentials: HashMap::new(), config: HashMap::new(), @@ -200,13 +206,16 @@ impl OpenShell for TestOpenShell { .provider .ok_or_else(|| Status::invalid_argument("provider is required"))?; let mut providers = self.state.providers.lock().await; - if providers.contains_key(&provider.name) { + let provider_name = provider.object_name().to_string(); + if providers.contains_key(&provider_name) { return Err(Status::already_exists("provider already exists")); } - if provider.id.is_empty() { - provider.id = format!("id-{}", provider.name); + if provider.object_id().is_empty() + && let Some(metadata) = &mut provider.metadata + { + metadata.id = format!("id-{provider_name}"); } - providers.insert(provider.name.clone(), provider.clone()); + providers.insert(provider_name, provider.clone()); Ok(Response::new(ProviderResponse { provider: Some(provider), })) @@ -242,6 +251,20 @@ impl OpenShell for TestOpenShell { Ok(Response::new(ListProvidersResponse { providers })) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, request: tonic::Request, @@ -253,13 +276,13 @@ impl OpenShell for TestOpenShell { let mut providers = self.state.providers.lock().await; let existing = providers - .get(&provider.name) + .get(provider.object_name()) .cloned() .ok_or_else(|| Status::not_found("provider not found"))?; // Merge semantics: empty map = no change, empty value = delete key. - let merge = |mut base: std::collections::HashMap, - incoming: std::collections::HashMap| - -> std::collections::HashMap { + let merge = |mut base: HashMap, + incoming: HashMap| + -> HashMap { if incoming.is_empty() { return base; } @@ -272,14 +295,21 @@ impl OpenShell for TestOpenShell { } base }; + let existing_metadata = existing.metadata.clone().unwrap_or_default(); + let provider_metadata = provider.metadata.clone().unwrap_or_default(); let updated = Provider { - id: existing.id, - name: provider.name, + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: existing_metadata.id, + name: provider_metadata.name, + created_at_ms: existing_metadata.created_at_ms, + labels: existing_metadata.labels, + }), r#type: existing.r#type, credentials: merge(existing.credentials, provider.credentials), config: merge(existing.config, provider.config), }; - providers.insert(updated.name.clone(), updated.clone()); + let updated_name = updated.object_name().to_string(); + providers.insert(updated_name, updated.clone()); Ok(Response::new(ProviderResponse { provider: Some(updated), })) @@ -298,6 +328,8 @@ impl OpenShell for TestOpenShell { tokio_stream::wrappers::ReceiverStream>; type ExecSandboxStream = tokio_stream::wrappers::ReceiverStream>; + type ConnectSupervisorStream = + tokio_stream::wrappers::ReceiverStream>; async fn watch_sandbox( &self, @@ -423,6 +455,23 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = + tokio_stream::wrappers::ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } // ── TLS helpers ────────────────────────────────────────────────────── diff --git a/crates/openshell-cli/tests/mtls_integration.rs b/crates/openshell-cli/tests/mtls_integration.rs index 5d04239bf5..0c76b34090 100644 --- a/crates/openshell-cli/tests/mtls_integration.rs +++ b/crates/openshell-cli/tests/mtls_integration.rs @@ -177,6 +177,20 @@ impl OpenShell for TestOpenShell { )) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, _request: tonic::Request, @@ -200,6 +214,9 @@ impl OpenShell for TestOpenShell { >; type ExecSandboxStream = tokio_stream::wrappers::ReceiverStream>; + type ConnectSupervisorStream = tokio_stream::wrappers::ReceiverStream< + Result, + >; async fn watch_sandbox( &self, @@ -325,6 +342,23 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = + tokio_stream::wrappers::ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } fn build_ca() -> (Certificate, KeyPair) { @@ -430,3 +464,44 @@ async fn cli_requires_client_cert_for_https() { let result = grpc_client(&endpoint, &tls).await; assert!(result.is_err()); } + +async fn run_server_no_client_auth( + server_cert: String, + server_key: String, +) -> std::net::SocketAddr { + let identity = Identity::from_pem(server_cert, server_key); + let tls = ServerTlsConfig::new().identity(identity); + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let incoming = TcpListenerStream::new(listener); + tokio::spawn(async move { + Server::builder() + .tls_config(tls) + .unwrap() + .add_service(OpenShellServer::new(TestOpenShell)) + .serve_with_incoming(incoming) + .await + .unwrap(); + }); + + addr +} + +#[tokio::test] +async fn cli_connects_with_gateway_insecure() { + install_rustls_provider(); + + let (ca, ca_key) = build_ca(); + let (server_cert, server_key) = build_server_cert(&ca, &ca_key); + + let addr = run_server_no_client_auth(server_cert, server_key).await; + + let mut tls = TlsOptions::default(); + tls.gateway_insecure = true; + + let endpoint = format!("https://localhost:{}", addr.port()); + let mut client = grpc_client(&endpoint, &tls).await.unwrap(); + let response = client.health(HealthRequest {}).await.unwrap(); + assert_eq!(response.get_ref().status, ServiceStatus::Healthy as i32); +} diff --git a/crates/openshell-cli/tests/provider_commands_integration.rs b/crates/openshell-cli/tests/provider_commands_integration.rs index c5476afee7..cdef9614e2 100644 --- a/crates/openshell-cli/tests/provider_commands_integration.rs +++ b/crates/openshell-cli/tests/provider_commands_integration.rs @@ -7,14 +7,16 @@ use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer}; use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - ExecSandboxEvent, ExecSandboxRequest, GetGatewayConfigRequest, GetGatewayConfigResponse, - GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, - GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxesRequest, ListSandboxesResponse, Provider, ProviderResponse, - RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, - ServiceStatus, UpdateProviderRequest, WatchSandboxRequest, + ExecSandboxEvent, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, + GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, + Provider, ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, + SandboxStreamEvent, ServiceStatus, SupervisorMessage, UpdateProviderRequest, + WatchSandboxRequest, }; +use openshell_core::{ObjectId, ObjectName}; use rcgen::{ BasicConstraints, Certificate, CertificateParams, ExtendedKeyUsagePurpose, IsCa, KeyPair, }; @@ -154,13 +156,16 @@ impl OpenShell for TestOpenShell { .provider .ok_or_else(|| Status::invalid_argument("provider is required"))?; let mut providers = self.state.providers.lock().await; - if providers.contains_key(&provider.name) { + let provider_name = provider.object_name().to_string(); + if providers.contains_key(&provider_name) { return Err(Status::already_exists("provider already exists")); } - if provider.id.is_empty() { - provider.id = format!("id-{}", provider.name); + if provider.object_id().is_empty() + && let Some(metadata) = &mut provider.metadata + { + metadata.id = format!("id-{provider_name}"); } - providers.insert(provider.name.clone(), provider.clone()); + providers.insert(provider_name, provider.clone()); Ok(Response::new(ProviderResponse { provider: Some(provider), })) @@ -196,6 +201,27 @@ impl OpenShell for TestOpenShell { Ok(Response::new(ListProvidersResponse { providers })) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Ok(Response::new( + openshell_core::proto::ListProviderProfilesResponse { + profiles: openshell_providers::default_profiles() + .iter() + .map(openshell_providers::ProviderTypeProfile::to_proto) + .collect(), + }, + )) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, request: tonic::Request, @@ -207,13 +233,13 @@ impl OpenShell for TestOpenShell { let mut providers = self.state.providers.lock().await; let existing = providers - .get(&provider.name) + .get(provider.object_name()) .cloned() .ok_or_else(|| Status::not_found("provider not found"))?; // Merge semantics: empty map = no change, empty value = delete key. - let merge = |mut base: std::collections::HashMap, - incoming: std::collections::HashMap| - -> std::collections::HashMap { + let merge = |mut base: HashMap, + incoming: HashMap| + -> HashMap { if incoming.is_empty() { return base; } @@ -226,14 +252,21 @@ impl OpenShell for TestOpenShell { } base }; + let existing_metadata = existing.metadata.clone().unwrap_or_default(); + let provider_metadata = provider.metadata.clone().unwrap_or_default(); let updated = Provider { - id: existing.id, - name: provider.name, + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: existing_metadata.id, + name: provider_metadata.name, + created_at_ms: existing_metadata.created_at_ms, + labels: existing_metadata.labels, + }), r#type: existing.r#type, credentials: merge(existing.credentials, provider.credentials), config: merge(existing.config, provider.config), }; - providers.insert(updated.name.clone(), updated.clone()); + let updated_name = updated.object_name().to_string(); + providers.insert(updated_name, updated.clone()); Ok(Response::new(ProviderResponse { provider: Some(updated), })) @@ -252,6 +285,8 @@ impl OpenShell for TestOpenShell { tokio_stream::wrappers::ReceiverStream>; type ExecSandboxStream = tokio_stream::wrappers::ReceiverStream>; + type ConnectSupervisorStream = + tokio_stream::wrappers::ReceiverStream>; async fn watch_sandbox( &self, @@ -377,6 +412,23 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = + tokio_stream::wrappers::ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } fn install_rustls_provider() { @@ -498,6 +550,15 @@ async fn provider_cli_run_functions_support_full_crud_flow() { .expect("provider delete"); } +#[tokio::test] +async fn provider_list_profiles_cli_uses_profile_browsing_rpc() { + let ts = run_server().await; + + run::provider_list_profiles(&ts.endpoint, &ts.tls) + .await + .expect("provider list-profiles"); +} + #[tokio::test] async fn provider_create_rejects_key_only_credentials_without_local_env_value() { let ts = run_server().await; diff --git a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs index d5d39f0821..b65bdd6844 100644 --- a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs +++ b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs @@ -8,18 +8,19 @@ use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer}; use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - ExecSandboxEvent, ExecSandboxRequest, GetGatewayConfigRequest, GetGatewayConfigResponse, - GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, - GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxesRequest, ListSandboxesResponse, PlatformEvent, ProviderResponse, - RevokeSshSessionRequest, RevokeSshSessionResponse, Sandbox, SandboxPhase, SandboxResponse, - SandboxStreamEvent, ServiceStatus, UpdateProviderRequest, WatchSandboxRequest, - sandbox_stream_event, + ExecSandboxEvent, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, + GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, + PlatformEvent, ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, Sandbox, + SandboxPhase, SandboxResponse, SandboxStreamEvent, ServiceStatus, SupervisorMessage, + UpdateProviderRequest, WatchSandboxRequest, sandbox_stream_event, }; use rcgen::{ BasicConstraints, Certificate, CertificateParams, ExtendedKeyUsagePurpose, IsCa, KeyPair, }; +use std::collections::HashMap; use std::fs; use std::os::unix::fs::PermissionsExt; use std::sync::Arc; @@ -112,9 +113,12 @@ impl OpenShell for TestOpenShell { Ok(Response::new(SandboxResponse { sandbox: Some(Sandbox { - id: format!("id-{sandbox_name}"), - name: sandbox_name, - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("id-{sandbox_name}"), + name: sandbox_name, + created_at_ms: 0, + labels: HashMap::new(), + }), phase: SandboxPhase::Provisioning as i32, ..Sandbox::default() }), @@ -128,9 +132,12 @@ impl OpenShell for TestOpenShell { let name = request.into_inner().name; Ok(Response::new(SandboxResponse { sandbox: Some(Sandbox { - id: format!("id-{name}"), - name, - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("id-{name}"), + name, + created_at_ms: 0, + labels: HashMap::new(), + }), phase: SandboxPhase::Ready as i32, ..Sandbox::default() }), @@ -224,6 +231,20 @@ impl OpenShell for TestOpenShell { Ok(Response::new(ListProvidersResponse::default())) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, _request: tonic::Request, @@ -242,6 +263,8 @@ impl OpenShell for TestOpenShell { tokio_stream::wrappers::ReceiverStream>; type ExecSandboxStream = tokio_stream::wrappers::ReceiverStream>; + type ConnectSupervisorStream = + tokio_stream::wrappers::ReceiverStream>; async fn watch_sandbox( &self, @@ -252,9 +275,12 @@ impl OpenShell for TestOpenShell { tokio::spawn(async move { let provisioning = Sandbox { - id: sandbox_id.clone(), - name: sandbox_id.trim_start_matches("id-").to_string(), - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: sandbox_id.clone(), + name: sandbox_id.trim_start_matches("id-").to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), phase: SandboxPhase::Provisioning as i32, ..Sandbox::default() }; @@ -403,6 +429,23 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = + tokio_stream::wrappers::ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } fn install_rustls_provider() { @@ -544,6 +587,7 @@ async fn sandbox_create_keeps_command_sessions_by_default() { None, None, None, + None, &[], None, None, @@ -551,6 +595,7 @@ async fn sandbox_create_keeps_command_sessions_by_default() { Some(false), Some(false), Some(false), + &HashMap::new(), &tls, ) .await @@ -584,6 +629,7 @@ async fn sandbox_create_deletes_command_sessions_with_no_keep() { None, None, None, + None, &[], None, None, @@ -591,6 +637,7 @@ async fn sandbox_create_deletes_command_sessions_with_no_keep() { Some(false), Some(false), Some(false), + &HashMap::new(), &tls, ) .await @@ -627,6 +674,7 @@ async fn sandbox_create_deletes_shell_sessions_with_no_keep() { None, None, None, + None, &[], None, None, @@ -634,6 +682,7 @@ async fn sandbox_create_deletes_shell_sessions_with_no_keep() { Some(true), Some(false), Some(false), + &HashMap::new(), &tls, ) .await @@ -670,6 +719,7 @@ async fn sandbox_create_keeps_sandbox_with_hidden_keep_flag() { None, None, None, + None, &[], None, None, @@ -677,6 +727,7 @@ async fn sandbox_create_keeps_sandbox_with_hidden_keep_flag() { Some(false), Some(false), Some(false), + &HashMap::new(), &tls, ) .await @@ -698,6 +749,10 @@ async fn sandbox_create_keeps_sandbox_with_forwarding() { let _env = test_env(&fake_ssh_dir, &xdg_dir); let tls = test_tls(&server); install_fake_ssh(&fake_ssh_dir); + let forward_port = { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + listener.local_addr().unwrap().port() + }; run::sandbox_create( &server.endpoint, @@ -710,13 +765,15 @@ async fn sandbox_create_keeps_sandbox_with_forwarding() { None, None, None, + None, &[], None, - Some(openshell_core::forward::ForwardSpec::new(8080)), + Some(openshell_core::forward::ForwardSpec::new(forward_port)), &["echo".to_string(), "OK".to_string()], Some(false), Some(false), Some(false), + &HashMap::new(), &tls, ) .await diff --git a/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs b/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs index fbadec4c34..bfad9a7d5f 100644 --- a/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs +++ b/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs @@ -8,12 +8,13 @@ use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer}; use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - ExecSandboxEvent, ExecSandboxRequest, GetGatewayConfigRequest, GetGatewayConfigResponse, - GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, - GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxesRequest, ListSandboxesResponse, ProviderResponse, Sandbox, SandboxResponse, - SandboxStreamEvent, ServiceStatus, UpdateProviderRequest, WatchSandboxRequest, + ExecSandboxEvent, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, + GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, + ProviderResponse, Sandbox, SandboxPolicy, SandboxResponse, SandboxStreamEvent, ServiceStatus, + SupervisorMessage, UpdateProviderRequest, WatchSandboxRequest, }; use rcgen::{ BasicConstraints, Certificate, CertificateParams, ExtendedKeyUsagePurpose, IsCa, KeyPair, @@ -110,9 +111,12 @@ impl OpenShell for TestOpenShell { *self.state.last_get_name.lock().await = Some(name.clone()); Ok(Response::new(SandboxResponse { sandbox: Some(Sandbox { - id: "test-id".to_string(), - name, - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "test-id".to_string(), + name, + created_at_ms: 0, + labels: std::collections::HashMap::new(), + }), ..Default::default() }), })) @@ -134,9 +138,20 @@ impl OpenShell for TestOpenShell { async fn get_sandbox_config( &self, - _request: tonic::Request, + request: tonic::Request, ) -> Result, Status> { - Ok(Response::new(GetSandboxConfigResponse::default())) + let req = request.into_inner(); + assert_eq!( + req.sandbox_id, "test-id", + "sandbox_get --policy-only should pass the id from GetSandbox" + ); + Ok(Response::new(GetSandboxConfigResponse { + policy: Some(SandboxPolicy { + version: 1, + ..Default::default() + }), + ..Default::default() + })) } async fn get_gateway_config( @@ -192,6 +207,20 @@ impl OpenShell for TestOpenShell { Ok(Response::new(ListProvidersResponse::default())) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, _request: tonic::Request, @@ -210,6 +239,8 @@ impl OpenShell for TestOpenShell { tokio_stream::wrappers::ReceiverStream>; type ExecSandboxStream = tokio_stream::wrappers::ReceiverStream>; + type ConnectSupervisorStream = + tokio_stream::wrappers::ReceiverStream>; async fn watch_sandbox( &self, @@ -335,6 +366,23 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = + tokio_stream::wrappers::ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } // ── helpers ─────────────────────────────────────────────────────────── @@ -432,7 +480,7 @@ async fn run_server() -> TestServer { async fn sandbox_get_sends_correct_name() { let ts = run_server().await; - run::sandbox_get(&ts.endpoint, "my-sandbox", &ts.tls) + run::sandbox_get(&ts.endpoint, "my-sandbox", false, &ts.tls) .await .expect("sandbox_get should succeed"); @@ -444,6 +492,19 @@ async fn sandbox_get_sends_correct_name() { ); } +/// `sandbox_get` with `policy_only` calls `GetSandboxConfig` and prints YAML from the response. +#[tokio::test] +async fn sandbox_get_policy_only_round_trip() { + let ts = run_server().await; + + run::sandbox_get(&ts.endpoint, "my-sandbox", true, &ts.tls) + .await + .expect("sandbox_get with policy_only should succeed"); + + let recorded = ts.openshell.state.last_get_name.lock().await.clone(); + assert_eq!(recorded.as_deref(), Some("my-sandbox")); +} + /// End-to-end: save a last-used sandbox, load it back, then call `sandbox_get` /// with the resolved name. This validates the persistence + gRPC wiring. #[tokio::test] @@ -462,7 +523,7 @@ async fn sandbox_get_with_persisted_last_sandbox() { assert_eq!(resolved, "persisted-sb"); // Call sandbox_get with the resolved name. - run::sandbox_get(&ts.endpoint, &resolved, &ts.tls) + run::sandbox_get(&ts.endpoint, &resolved, false, &ts.tls) .await .expect("sandbox_get should succeed"); @@ -484,7 +545,7 @@ async fn explicit_name_takes_precedence_over_persisted() { // Persist one name, but supply a different one explicitly. save_last_sandbox("my-cluster", "old-sandbox").expect("save should succeed"); - run::sandbox_get(&ts.endpoint, "explicit-sandbox", &ts.tls) + run::sandbox_get(&ts.endpoint, "explicit-sandbox", false, &ts.tls) .await .expect("sandbox_get should succeed"); diff --git a/crates/openshell-core/build.rs b/crates/openshell-core/build.rs index c89a03483c..7613c8754b 100644 --- a/crates/openshell-core/build.rs +++ b/crates/openshell-core/build.rs @@ -2,6 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 use std::env; +use std::path::{Path, PathBuf}; + +const PROTO_REL: &str = "../../proto"; fn main() -> Result<(), Box> { // --- Git-derived version --- @@ -17,6 +20,8 @@ fn main() -> Result<(), Box> { } // --- Protobuf compilation --- + // Re-run when anything under proto/ changes (including newly added .proto files). + println!("cargo:rerun-if-changed={PROTO_REL}"); // Use bundled protoc from protobuf-src. The system protoc (from apt-get) // does not bundle the well-known type includes (google/protobuf/struct.proto // etc.), so we must use protobuf-src which ships both the binary and the @@ -28,26 +33,31 @@ fn main() -> Result<(), Box> { env::set_var("PROTOC", protobuf_src::protoc()); } - let proto_files = [ - "../../proto/openshell.proto", - "../../proto/datamodel.proto", - "../../proto/sandbox.proto", - "../../proto/compute_driver.proto", - "../../proto/inference.proto", - "../../proto/test.proto", - ]; + let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR")?); + let proto_root = manifest_dir.join(PROTO_REL); + + let mut proto_files = Vec::new(); + collect_proto_files(&proto_root, &mut proto_files)?; + proto_files.sort(); // Configure tonic-build tonic_build::configure() .build_server(true) .build_client(true) - .compile_protos(&proto_files, &["../../proto"])?; + .compile_protos(&proto_files, &[proto_root.as_path()])?; - // Tell cargo to rerun if the proto file changes - for proto_file in proto_files { - println!("cargo:rerun-if-changed={proto_file}"); - } + Ok(()) +} +fn collect_proto_files(dir: &Path, out: &mut Vec) -> std::io::Result<()> { + for entry in std::fs::read_dir(dir)? { + let path = entry?.path(); + if path.is_dir() { + collect_proto_files(&path, out)?; + } else if path.extension().is_some_and(|ext| ext == "proto") { + out.push(path); + } + } Ok(()) } @@ -63,8 +73,14 @@ fn main() -> Result<(), Box> { /// /// Returns `None` when git is unavailable or the repo has no matching tags. fn git_version() -> Option { + // Match numeric release tags only (e.g. `v0.0.29`). The bare glob `v*` + // also matches non-release tags like `vm-dev` or `vm-prod`; when one of + // those lands on the same commit as a release tag, `git describe` picks + // it and the resulting version string collapses to `m-dev` after the + // leading `v` is stripped below. Requiring a digit after `v` excludes + // those development tags without losing any release tag. let output = std::process::Command::new("git") - .args(["describe", "--tags", "--long", "--match", "v*"]) + .args(["describe", "--tags", "--long", "--match", "v[0-9]*"]) .output() .ok()?; diff --git a/crates/openshell-core/src/config.rs b/crates/openshell-core/src/config.rs index 279752b4fd..1ec06677b3 100644 --- a/crates/openshell-core/src/config.rs +++ b/crates/openshell-core/src/config.rs @@ -1,19 +1,58 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Configuration management for OpenShell components. +//! Configuration management for `OpenShell` components. use serde::{Deserialize, Serialize}; use std::fmt; use std::net::SocketAddr; use std::path::PathBuf; +use std::process::Command; use std::str::FromStr; +// ── Public default constants ──────────────────────────────────────────── +// +// Canonical source for default values used across multiple crates. +// Clap `default_value_t` annotations and runtime fallbacks should +// reference these constants instead of hardcoding literals. + +/// Default SSH port inside sandbox containers. +pub const DEFAULT_SSH_PORT: u16 = 2222; + +/// Default server / SSH gateway port. +pub const DEFAULT_SERVER_PORT: u16 = 8080; + +/// Default container stop timeout in seconds (SIGTERM → SIGKILL). +pub const DEFAULT_STOP_TIMEOUT_SECS: u32 = 10; + +/// Default allowed clock skew for SSH handshake validation, in seconds. +pub const DEFAULT_SSH_HANDSHAKE_SKEW_SECS: u64 = 300; + +/// Default Podman bridge network name. +pub const DEFAULT_NETWORK_NAME: &str = "openshell"; + +/// Default Docker bridge network name for local sandboxes. +pub const DEFAULT_DOCKER_NETWORK_NAME: &str = "openshell-docker"; + +/// Default OCI image for the openshell-sandbox supervisor binary. +pub const DEFAULT_SUPERVISOR_IMAGE: &str = "openshell/supervisor:latest"; + +/// Default image pull policy for sandbox images. +pub const DEFAULT_IMAGE_PULL_POLICY: &str = "missing"; + +/// Default Kubernetes namespace for sandbox resources. +pub const DEFAULT_K8S_NAMESPACE: &str = "openshell"; + +/// CDI device identifier for requesting all NVIDIA GPUs. +pub const CDI_GPU_DEVICE_ALL: &str = "nvidia.com/gpu=all"; + /// Compute backends the gateway can orchestrate sandboxes through. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum ComputeDriverKind { Kubernetes, + Vm, + Docker, Podman, } @@ -22,6 +61,8 @@ impl ComputeDriverKind { pub const fn as_str(self) -> &'static str { match self { Self::Kubernetes => "kubernetes", + Self::Vm => "vm", + Self::Docker => "docker", Self::Podman => "podman", } } @@ -39,14 +80,50 @@ impl FromStr for ComputeDriverKind { fn from_str(value: &str) -> Result { match value.trim().to_ascii_lowercase().as_str() { "kubernetes" => Ok(Self::Kubernetes), + "vm" => Ok(Self::Vm), + "docker" => Ok(Self::Docker), "podman" => Ok(Self::Podman), other => Err(format!( - "unsupported compute driver '{other}'. expected one of: kubernetes, podman" + "unsupported compute driver '{other}'. expected one of: kubernetes, vm, docker, podman" )), } } } +/// Auto-detect the appropriate compute driver based on the runtime environment. +/// +/// Priority order: Kubernetes → Podman → Docker. +/// VM is never auto-detected (requires explicit `--drivers vm`). +/// +/// Returns the first driver where the environment check passes. +/// Returns `None` if no compatible driver is found. +pub fn detect_driver() -> Option { + // Kubernetes: check for KUBERNETES_SERVICE_HOST env var (set inside pods) + if std::env::var_os("KUBERNETES_SERVICE_HOST").is_some() { + return Some(ComputeDriverKind::Kubernetes); + } + + // Podman: check if podman binary is available + if is_binary_available("podman") { + return Some(ComputeDriverKind::Podman); + } + + // Docker: check if docker binary is available + if is_binary_available("docker") { + return Some(ComputeDriverKind::Docker); + } + + None +} + +/// Check if a binary is available on the system PATH. +fn is_binary_available(name: &str) -> bool { + Command::new(name) + .arg("--version") + .output() + .is_ok_and(|output| output.status.success()) +} + /// Server configuration. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct Config { @@ -54,6 +131,27 @@ pub struct Config { #[serde(default = "default_bind_address")] pub bind_address: SocketAddr, + /// Address to bind the unauthenticated health endpoint to. + /// + /// When `None`, the dedicated health listener is disabled. + #[serde(default)] + pub health_bind_address: Option, + + /// Address to bind the Prometheus metrics endpoint to. + /// + /// When `None`, the dedicated metrics listener is disabled. + #[serde(default)] + pub metrics_bind_address: Option, + + /// Additional bind addresses that serve the same multiplexed gRPC/HTTP + /// surface as `bind_address`. + /// + /// Compute drivers may register extra listeners during startup so that + /// sandbox workloads can call back into the gateway over an interface + /// that the operator-supplied `bind_address` does not expose. + #[serde(default)] + pub extra_bind_addresses: Vec, + /// Log level (trace, debug, info, warn, error). #[serde(default = "default_log_level")] pub log_level: String, @@ -61,6 +159,10 @@ pub struct Config { /// TLS configuration. When `None`, the server listens on plaintext HTTP. pub tls: Option, + /// OIDC configuration. When `Some`, the server validates Bearer JWTs. + #[serde(default)] + pub oidc: Option, + /// Database URL for persistence. pub database_url: String, @@ -69,7 +171,7 @@ pub struct Config { /// The config shape allows multiple drivers so the gateway can evolve /// toward multi-backend routing. Current releases require exactly one /// configured driver. - #[serde(default = "default_compute_drivers")] + #[serde(default)] pub compute_drivers: Vec, /// Kubernetes namespace for sandboxes. @@ -77,7 +179,7 @@ pub struct Config { pub sandbox_namespace: String, /// Default container image for sandboxes. - #[serde(default)] + #[serde(default = "default_sandbox_image")] pub sandbox_image: String, /// Kubernetes `imagePullPolicy` for sandbox pods (e.g. `Always`, @@ -87,7 +189,7 @@ pub struct Config { #[serde(default)] pub sandbox_image_pull_policy: String, - /// gRPC endpoint for sandboxes to connect back to OpenShell. + /// gRPC endpoint for sandboxes to connect back to `OpenShell`. /// Used by sandbox pods to fetch their policy at startup. #[serde(default)] pub grpc_endpoint: String, @@ -104,10 +206,22 @@ pub struct Config { #[serde(default = "default_ssh_connect_path")] pub ssh_connect_path: String, - /// SSH listen port inside sandbox pods. + /// SSH listen port inside sandbox containers that expose a TCP endpoint. #[serde(default = "default_sandbox_ssh_port")] pub sandbox_ssh_port: u16, + /// Filesystem path where the sandbox supervisor binds its SSH Unix + /// socket. The supervisor is passed this path via + /// `OPENSHELL_SSH_SOCKET_PATH` / `--ssh-socket-path` and connects its + /// relay bridge to the same path. + /// + /// When the gateway orchestrates sandboxes that each live in their own + /// filesystem (K8s pod, libkrun VM, etc.), the default is safe. For + /// local dev where multiple supervisors share `/run`, override this to + /// something unique per sandbox. + #[serde(default = "default_sandbox_ssh_socket_path")] + pub sandbox_ssh_socket_path: String, + /// Shared secret for gateway-to-sandbox SSH handshake. #[serde(default)] pub ssh_handshake_secret: String, @@ -160,23 +274,86 @@ pub struct TlsConfig { pub allow_unauthenticated: bool, } +/// OIDC (`OpenID` Connect) configuration for JWT-based authentication. +/// +/// When configured, the server validates `authorization: Bearer ` +/// headers on gRPC requests against the specified issuer's JWKS endpoint. +/// +/// The roles claim path is configurable to support different providers: +/// - Keycloak: `realm_access.roles` (default) +/// - Entra ID / Okta: `roles` +/// - Custom: any dot-separated path into the JWT claims +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct OidcConfig { + /// OIDC issuer URL (e.g., `http://localhost:8180/realms/openshell`). + pub issuer: String, + + /// Expected audience (`aud`) claim. Typically the OIDC client ID. + pub audience: String, + + /// JWKS cache TTL in seconds. Defaults to 3600 (1 hour). + #[serde(default = "default_jwks_ttl_secs")] + pub jwks_ttl_secs: u64, + + /// Dot-separated path to the roles array in the JWT claims. + /// Defaults to `realm_access.roles` (Keycloak). + /// Examples: `roles` (Entra ID), `groups` (Okta), `custom.path.roles`. + #[serde(default = "default_roles_claim")] + pub roles_claim: String, + + /// Role name that grants admin access. Defaults to `openshell-admin`. + #[serde(default = "default_admin_role")] + pub admin_role: String, + + /// Role name that grants standard user access. Defaults to `openshell-user`. + #[serde(default = "default_user_role")] + pub user_role: String, + + /// Dot-separated path to the scopes value in the JWT claims. + /// When non-empty, the server enforces scope-based permissions on top of roles. + /// Keycloak: `scope` (space-delimited string). Okta: `scp` (JSON array). + #[serde(default)] + pub scopes_claim: String, +} + +const fn default_jwks_ttl_secs() -> u64 { + 3600 +} + +fn default_roles_claim() -> String { + "realm_access.roles".to_string() +} + +fn default_admin_role() -> String { + "openshell-admin".to_string() +} + +fn default_user_role() -> String { + "openshell-user".to_string() +} + impl Config { /// Create a new config with optional TLS. pub fn new(tls: Option) -> Self { Self { bind_address: default_bind_address(), + health_bind_address: None, + metrics_bind_address: None, + extra_bind_addresses: Vec::new(), log_level: default_log_level(), tls, + oidc: None, database_url: String::new(), - compute_drivers: default_compute_drivers(), + compute_drivers: vec![], sandbox_namespace: default_sandbox_namespace(), - sandbox_image: String::new(), + sandbox_image: default_sandbox_image(), sandbox_image_pull_policy: String::new(), grpc_endpoint: String::new(), ssh_gateway_host: default_ssh_gateway_host(), ssh_gateway_port: default_ssh_gateway_port(), ssh_connect_path: default_ssh_connect_path(), sandbox_ssh_port: default_sandbox_ssh_port(), + sandbox_ssh_socket_path: default_sandbox_ssh_socket_path(), ssh_handshake_secret: String::new(), ssh_handshake_skew_secs: default_ssh_handshake_skew_secs(), ssh_session_ttl_secs: default_ssh_session_ttl_secs(), @@ -192,6 +369,31 @@ impl Config { self } + #[must_use] + pub const fn with_health_bind_address(mut self, addr: SocketAddr) -> Self { + self.health_bind_address = Some(addr); + self + } + + #[must_use] + pub const fn with_metrics_bind_address(mut self, addr: SocketAddr) -> Self { + self.metrics_bind_address = Some(addr); + self + } + + /// Append an extra listener address to the multiplex service. + /// + /// Duplicate entries (matching `bind_address` or any existing entry) are + /// silently dropped so callers can naively push driver-derived addresses + /// without checking for collisions. + #[must_use] + pub fn with_extra_bind_address(mut self, addr: SocketAddr) -> Self { + if addr != self.bind_address && !self.extra_bind_addresses.contains(&addr) { + self.extra_bind_addresses.push(addr); + } + self + } + /// Create a new configuration with the given log level. #[must_use] pub fn with_log_level(mut self, level: impl Into) -> Self { @@ -306,10 +508,17 @@ impl Config { self.host_gateway_ip = ip.into(); self } + + /// Set the OIDC configuration for JWT-based authentication. + #[must_use] + pub fn with_oidc(mut self, oidc: OidcConfig) -> Self { + self.oidc = Some(oidc); + self + } } fn default_bind_address() -> SocketAddr { - "0.0.0.0:8080".parse().expect("valid default address") + "127.0.0.1:8080".parse().expect("valid default address") } fn default_log_level() -> String { @@ -320,8 +529,8 @@ fn default_sandbox_namespace() -> String { "default".to_string() } -fn default_compute_drivers() -> Vec { - vec![ComputeDriverKind::Kubernetes] +fn default_sandbox_image() -> String { + format!("{}/base:latest", crate::image::DEFAULT_COMMUNITY_REGISTRY) } fn default_ssh_gateway_host() -> String { @@ -329,19 +538,23 @@ fn default_ssh_gateway_host() -> String { } const fn default_ssh_gateway_port() -> u16 { - 8080 + DEFAULT_SERVER_PORT } fn default_ssh_connect_path() -> String { "/connect/ssh".to_string() } +fn default_sandbox_ssh_socket_path() -> String { + "/run/openshell/ssh.sock".to_string() +} + const fn default_sandbox_ssh_port() -> u16 { - 2222 + DEFAULT_SSH_PORT } const fn default_ssh_handshake_skew_secs() -> u64 { - 300 + DEFAULT_SSH_HANDSHAKE_SKEW_SECS } const fn default_ssh_session_ttl_secs() -> u64 { @@ -350,7 +563,8 @@ const fn default_ssh_session_ttl_secs() -> u64 { #[cfg(test)] mod tests { - use super::{ComputeDriverKind, Config}; + use super::{ComputeDriverKind, Config, detect_driver}; + use std::net::SocketAddr; #[test] fn compute_driver_kind_parses_supported_values() { @@ -358,23 +572,74 @@ mod tests { "kubernetes".parse::().unwrap(), ComputeDriverKind::Kubernetes ); + assert_eq!( + "vm".parse::().unwrap(), + ComputeDriverKind::Vm + ); assert_eq!( "podman".parse::().unwrap(), ComputeDriverKind::Podman ); + assert_eq!( + "docker".parse::().unwrap(), + ComputeDriverKind::Docker + ); } #[test] fn compute_driver_kind_rejects_unknown_values() { - let err = "docker".parse::().unwrap_err(); - assert!(err.contains("unsupported compute driver 'docker'")); + let err = "firecracker".parse::().unwrap_err(); + assert!(err.contains("unsupported compute driver 'firecracker'")); } #[test] - fn config_defaults_to_kubernetes_driver() { - assert_eq!( - Config::new(None).compute_drivers, - vec![ComputeDriverKind::Kubernetes] - ); + fn config_defaults_to_loopback_bind_address() { + let expected: SocketAddr = "127.0.0.1:8080".parse().expect("valid address"); + assert_eq!(Config::new(None).bind_address, expected); + } + + #[test] + fn config_new_disables_health_bind_by_default() { + let cfg = Config::new(None); + assert!(cfg.health_bind_address.is_none()); + } + + #[test] + fn config_with_health_bind_address_sets_address() { + let addr: SocketAddr = "0.0.0.0:9090".parse().expect("valid address"); + let cfg = Config::new(None).with_health_bind_address(addr); + assert_eq!(cfg.health_bind_address, Some(addr)); + } + + #[test] + fn detect_driver_returns_none_without_k8s_env_or_binaries() { + // When KUBERNETES_SERVICE_HOST is not set and no docker/podman binaries + // are available, detect_driver should return None. + // This test may pass or fail depending on the test environment, + // but it documents the expected behavior. + let _ = detect_driver(); // Returns Some or None based on environment + } + + #[test] + #[allow(unsafe_code)] // std::env::set_var/remove_var require unsafe in Rust 2024 + fn detect_driver_prefers_kubernetes_when_k8s_env_is_set() { + // Save the original env var + let original = std::env::var("KUBERNETES_SERVICE_HOST").ok(); + + // Set the env var + unsafe { + std::env::set_var("KUBERNETES_SERVICE_HOST", "127.0.0.1"); + } + + let result = detect_driver(); + assert_eq!(result, Some(ComputeDriverKind::Kubernetes)); + + // Restore the original env var + unsafe { + match original { + Some(val) => std::env::set_var("KUBERNETES_SERVICE_HOST", val), + None => std::env::remove_var("KUBERNETES_SERVICE_HOST"), + } + } } } diff --git a/crates/openshell-core/src/error.rs b/crates/openshell-core/src/error.rs index 2399368fc8..7c33c9eaf8 100644 --- a/crates/openshell-core/src/error.rs +++ b/crates/openshell-core/src/error.rs @@ -1,15 +1,15 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Common error types for OpenShell. +//! Common error types for `OpenShell`. use miette::Diagnostic; use thiserror::Error; -/// Result type alias using OpenShell's error type. +/// Result type alias using `OpenShell`'s error type. pub type Result = std::result::Result; -/// OpenShell error type. +/// `OpenShell` error type. #[derive(Debug, Error, Diagnostic)] pub enum Error { /// Configuration error. @@ -103,3 +103,20 @@ impl Error { } } } + +/// Error type shared by all compute driver implementations. +/// +/// Both the Podman and Kubernetes drivers map their backend-specific +/// errors into these variants before crossing crate boundaries. +#[derive(Debug, Error)] +pub enum ComputeDriverError { + /// The requested sandbox already exists. + #[error("sandbox already exists")] + AlreadyExists, + /// A precondition for the operation was not met. + #[error("{0}")] + Precondition(String), + /// Generic error message. + #[error("{0}")] + Message(String), +} diff --git a/crates/openshell-core/src/forward.rs b/crates/openshell-core/src/forward.rs index c7b63fefb7..b48e5594a4 100644 --- a/crates/openshell-core/src/forward.rs +++ b/crates/openshell-core/src/forward.rs @@ -135,18 +135,17 @@ pub fn pid_matches_forward(pid: u32, port: u16, sandbox_id: Option<&str>) -> boo /// match is expected. pub fn find_forward_by_port(port: u16) -> Result> { let dir = forward_pid_dir()?; - let entries = match std::fs::read_dir(&dir) { - Ok(e) => e, - Err(_) => return Ok(None), + let Ok(entries) = std::fs::read_dir(&dir) else { + return Ok(None); }; let suffix = format!("-{port}.pid"); for entry in entries.flatten() { let file_name = entry.file_name(); let file_name = file_name.to_string_lossy(); - if let Some(name) = file_name.strip_suffix(&suffix) { - if !name.is_empty() { - return Ok(Some(name.to_string())); - } + if let Some(name) = file_name.strip_suffix(&suffix) + && !name.is_empty() + { + return Ok(Some(name.to_string())); } } Ok(None) @@ -487,6 +486,188 @@ pub fn shell_escape(value: &str) -> String { format!("'{escaped}'") } +/// Build the SSH `ProxyCommand` string used to tunnel to a sandbox. +/// +/// Every interpolated argument is shell-escaped so that server-supplied values +/// (gateway URL, sandbox id, token, gateway name) cannot inject shell +/// metacharacters into the command that OpenSSH executes via `/bin/sh -c`. +pub fn build_proxy_command( + exe: &str, + gateway_url: &str, + sandbox_id: &str, + token: &str, + gateway_name: &str, +) -> String { + format!( + "{} ssh-proxy --gateway {} --sandbox-id {} --token {} --gateway-name {}", + shell_escape(exe), + shell_escape(gateway_url), + shell_escape(sandbox_id), + shell_escape(token), + shell_escape(gateway_name), + ) +} + +/// Error returned when a `CreateSshSessionResponse` fails validation. +/// +/// The response fields flow into a `ProxyCommand` string executed by +/// `/bin/sh -c`; any deviation from the documented charset is rejected at the +/// gRPC trust boundary before escaping is attempted. +#[derive(Debug, thiserror::Error)] +pub enum SshSessionResponseError { + #[error("{field} is empty")] + Empty { field: &'static str }, + #[error("{field} exceeds maximum length of {max} bytes")] + TooLong { field: &'static str, max: usize }, + #[error("{field} contains invalid characters")] + InvalidChars { field: &'static str }, + #[error("gateway_scheme must be 'http' or 'https'")] + InvalidScheme, + #[error("gateway_port must be in range 1..=65535")] + InvalidPort, + #[error("connect_path must start with '/'")] + ConnectPathNotAbsolute, +} + +const MAX_SANDBOX_ID_LEN: usize = 128; +const MAX_TOKEN_LEN: usize = 4096; +const MAX_GATEWAY_HOST_LEN: usize = 253; +const MAX_CONNECT_PATH_LEN: usize = 2048; +const MAX_FINGERPRINT_LEN: usize = 256; + +fn is_sandbox_id_byte(b: u8) -> bool { + b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_') +} + +fn is_token_byte(b: u8) -> bool { + // URL-safe base64 + common token charset. No shell metacharacters, no + // whitespace, no control bytes. + b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_' | b'~' | b'+' | b'/' | b'=') +} + +fn is_gateway_host_byte(b: u8) -> bool { + // DNS hostname (alphanumeric + `.-`), IPv4, or bracketed IPv6 (`[::1]`). + // Rejects Unicode — callers must Punycode-encode IDN hosts before emitting. + b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b':' | b'[' | b']') +} + +fn is_connect_path_byte(b: u8) -> bool { + // RFC 3986 path charset (pchar) without `?`, `#`, space, backtick, or + // backslash. `%` is permitted so percent-encoded segments round-trip. + b.is_ascii_alphanumeric() + || matches!( + b, + b'-' | b'.' + | b'_' + | b'~' + | b'!' + | b'$' + | b'&' + | b'\'' + | b'(' + | b')' + | b'*' + | b'+' + | b',' + | b';' + | b'=' + | b':' + | b'@' + | b'/' + | b'%' + ) +} + +fn is_fingerprint_byte(b: u8) -> bool { + b.is_ascii_alphanumeric() || matches!(b, b':' | b'+' | b'/' | b'=' | b'-') +} + +/// Validate a `CreateSshSessionResponse` before any of its fields are used to +/// build a shell command or config file. +/// +/// This is a belt-and-suspenders pair to [`build_proxy_command`]: escaping +/// alone is sufficient to prevent injection, but rejecting malformed fields +/// at the trust boundary fails loudly before the string is assembled and +/// catches gateway bugs or tampering early. +pub fn validate_ssh_session_response( + resp: &crate::proto::CreateSshSessionResponse, +) -> std::result::Result<(), SshSessionResponseError> { + validate_field( + "sandbox_id", + &resp.sandbox_id, + MAX_SANDBOX_ID_LEN, + is_sandbox_id_byte, + )?; + validate_field("token", &resp.token, MAX_TOKEN_LEN, is_token_byte)?; + validate_field( + "gateway_host", + &resp.gateway_host, + MAX_GATEWAY_HOST_LEN, + is_gateway_host_byte, + )?; + match resp.gateway_scheme.as_str() { + "http" | "https" => {} + _ => return Err(SshSessionResponseError::InvalidScheme), + } + if resp.gateway_port == 0 || resp.gateway_port > u32::from(u16::MAX) { + return Err(SshSessionResponseError::InvalidPort); + } + if resp.connect_path.is_empty() { + return Err(SshSessionResponseError::Empty { + field: "connect_path", + }); + } + if !resp.connect_path.starts_with('/') { + return Err(SshSessionResponseError::ConnectPathNotAbsolute); + } + if resp.connect_path.len() > MAX_CONNECT_PATH_LEN { + return Err(SshSessionResponseError::TooLong { + field: "connect_path", + max: MAX_CONNECT_PATH_LEN, + }); + } + if !resp.connect_path.bytes().all(is_connect_path_byte) { + return Err(SshSessionResponseError::InvalidChars { + field: "connect_path", + }); + } + if !resp.host_key_fingerprint.is_empty() { + if resp.host_key_fingerprint.len() > MAX_FINGERPRINT_LEN { + return Err(SshSessionResponseError::TooLong { + field: "host_key_fingerprint", + max: MAX_FINGERPRINT_LEN, + }); + } + if !resp.host_key_fingerprint.bytes().all(is_fingerprint_byte) { + return Err(SshSessionResponseError::InvalidChars { + field: "host_key_fingerprint", + }); + } + } + Ok(()) +} + +fn validate_field( + name: &'static str, + value: &str, + max_len: usize, + byte_ok: fn(u8) -> bool, +) -> std::result::Result<(), SshSessionResponseError> { + if value.is_empty() { + return Err(SshSessionResponseError::Empty { field: name }); + } + if value.len() > max_len { + return Err(SshSessionResponseError::TooLong { + field: name, + max: max_len, + }); + } + if !value.bytes().all(byte_ok) { + return Err(SshSessionResponseError::InvalidChars { field: name }); + } + Ok(()) +} + /// Build notes string for a sandbox based on active forwards. /// /// Returns a string like `fwd:8080,3000` or an empty string if no forwards @@ -569,6 +750,202 @@ mod tests { assert_eq!(shell_escape("it's"), "'it'\"'\"'s'"); } + fn valid_session_response() -> crate::proto::CreateSshSessionResponse { + crate::proto::CreateSshSessionResponse { + sandbox_id: "sb-1234".to_string(), + token: "abcDEF-123_456.789".to_string(), + gateway_scheme: "https".to_string(), + gateway_host: "gateway.example.com".to_string(), + gateway_port: 443, + connect_path: "/connect/ssh".to_string(), + host_key_fingerprint: String::new(), + expires_at_ms: 0, + } + } + + #[test] + fn validate_ssh_session_response_accepts_realistic_response() { + assert!(validate_ssh_session_response(&valid_session_response()).is_ok()); + } + + #[test] + fn validate_ssh_session_response_accepts_bracketed_ipv6_host() { + let mut r = valid_session_response(); + r.gateway_host = "[::1]".to_string(); + assert!(validate_ssh_session_response(&r).is_ok()); + } + + #[test] + fn validate_ssh_session_response_accepts_optional_fingerprint() { + let mut r = valid_session_response(); + r.host_key_fingerprint = "SHA256:abcd+/=".to_string(); + assert!(validate_ssh_session_response(&r).is_ok()); + } + + #[test] + fn validate_ssh_session_response_rejects_empty_sandbox_id() { + let mut r = valid_session_response(); + r.sandbox_id.clear(); + assert!(matches!( + validate_ssh_session_response(&r), + Err(SshSessionResponseError::Empty { + field: "sandbox_id" + }) + )); + } + + #[test] + fn validate_ssh_session_response_rejects_shell_metachars_in_sandbox_id() { + for bad in ["a;b", "a b", "a$(id)", "a`id`", "a|b", "a&b", "a\nb"] { + let mut r = valid_session_response(); + r.sandbox_id = bad.to_string(); + assert!( + validate_ssh_session_response(&r).is_err(), + "expected reject for sandbox_id={bad:?}" + ); + } + } + + #[test] + fn validate_ssh_session_response_rejects_shell_metachars_in_token() { + for bad in ["$(id)", "`id`", "a;b", "a b", "a\tb", "a\0b"] { + let mut r = valid_session_response(); + r.token = bad.to_string(); + assert!( + validate_ssh_session_response(&r).is_err(), + "expected reject for token={bad:?}" + ); + } + } + + #[test] + fn validate_ssh_session_response_rejects_invalid_gateway_host() { + for bad in ["evil; cmd", "evil host", "ev$(id)il", "ev\nil", "evil/x"] { + let mut r = valid_session_response(); + r.gateway_host = bad.to_string(); + assert!( + validate_ssh_session_response(&r).is_err(), + "expected reject for gateway_host={bad:?}" + ); + } + } + + #[test] + fn validate_ssh_session_response_rejects_unknown_scheme() { + for bad in ["javascript", "file", "", "HTTPS", "ftp"] { + let mut r = valid_session_response(); + r.gateway_scheme = bad.to_string(); + assert!( + matches!( + validate_ssh_session_response(&r), + Err(SshSessionResponseError::InvalidScheme) + ), + "expected InvalidScheme for scheme={bad:?}" + ); + } + } + + #[test] + fn validate_ssh_session_response_rejects_out_of_range_port() { + for bad in [0u32, 65_536, 100_000] { + let mut r = valid_session_response(); + r.gateway_port = bad; + assert!(matches!( + validate_ssh_session_response(&r), + Err(SshSessionResponseError::InvalidPort) + )); + } + } + + #[test] + fn validate_ssh_session_response_rejects_connect_path_without_leading_slash() { + let mut r = valid_session_response(); + r.connect_path = "connect/ssh".to_string(); + assert!(matches!( + validate_ssh_session_response(&r), + Err(SshSessionResponseError::ConnectPathNotAbsolute) + )); + } + + #[test] + fn validate_ssh_session_response_rejects_injected_connect_path() { + // `$`, `(`, `)` are valid RFC 3986 sub-delims (pchar) so the validator + // permits them; shell_escape is the second defensive layer. The + // following characters are rejected at the validator boundary because + // they are either unambiguously hostile in a shell context or invalid + // per RFC 3986 in the path component. + for bad in ["/x`id`y", "/x y", "/x\nb", "/x\\b", "/x?q=1", "/x#frag"] { + let mut r = valid_session_response(); + r.connect_path = bad.to_string(); + assert!( + validate_ssh_session_response(&r).is_err(), + "expected reject for connect_path={bad:?}" + ); + } + } + + #[test] + fn build_proxy_command_escapes_shell_metacharacters() { + // Attacker-controlled values in every escapable position. + let cmd = build_proxy_command( + "/usr/local/bin/openshell", + "https://gw:443/connect", + "x$(touch /tmp/pwn)x", + "tok`id`", + "gw-name", + ); + + // The `$` / backtick must only appear inside single-quoted regions. + // A simple grep-based check: split on single-quoted runs and assert + // no shell metacharacter remains in the unquoted remainder. + assert!(!outside_single_quotes(&cmd).contains('$')); + assert!(!outside_single_quotes(&cmd).contains('`')); + assert!(!outside_single_quotes(&cmd).contains('|')); + assert!(!outside_single_quotes(&cmd).contains(';')); + assert!(!outside_single_quotes(&cmd).contains('&')); + assert!(!outside_single_quotes(&cmd).contains('\n')); + } + + #[test] + fn build_proxy_command_empty_values_quote_rather_than_vanish() { + // An empty value must become `''` rather than disappearing — otherwise + // downstream argv splitting would misalign. + let cmd = build_proxy_command("exe", "gw", "", "tok", "name"); + assert!(cmd.contains("--sandbox-id ''")); + } + + #[test] + fn build_proxy_command_safe_values_pass_through_unquoted() { + let cmd = build_proxy_command( + "/usr/local/bin/openshell", + "gw", + "sb-123", + "tok.456", + "name_1", + ); + assert_eq!( + cmd, + "/usr/local/bin/openshell ssh-proxy --gateway gw --sandbox-id sb-123 --token tok.456 --gateway-name name_1" + ); + } + + /// Helper: return the concatenation of characters that appear outside + /// POSIX single-quoted runs. Used by the metacharacter assertions above. + fn outside_single_quotes(s: &str) -> String { + let mut out = String::new(); + let mut inside = false; + for c in s.chars() { + if c == '\'' { + inside = !inside; + continue; + } + if !inside { + out.push(c); + } + } + out + } + #[test] fn build_sandbox_notes_with_forwards() { let forwards = vec![ @@ -642,12 +1019,28 @@ mod tests { #[test] fn check_port_available_free_port() { // Bind to port 0 to get an OS-assigned free port, then drop the - // listener so the port is released before we test it. - let listener = TcpListener::bind("127.0.0.1:0").unwrap(); - let port = listener.local_addr().unwrap().port(); - drop(listener); + // listener so the port is released before we test it. On busy CI + // hosts, another process can claim that single ephemeral port before + // we re-bind it, so retry with fresh OS-assigned ports. + let mut last_error = None; + for _ in 0..20 { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + drop(listener); + + match check_port_available(&ForwardSpec::new(port)) { + Ok(()) => return, + Err(err) => { + last_error = Some(err.to_string()); + std::thread::sleep(std::time::Duration::from_millis(10)); + } + } + } - assert!(check_port_available(&ForwardSpec::new(port)).is_ok()); + panic!( + "expected an OS-assigned port to be available; last error: {}", + last_error.unwrap_or_else(|| "none".to_string()) + ); } #[test] @@ -671,9 +1064,8 @@ mod tests { // `python3 -m http.server` which listens on [::] by default. The // IPv4-only TcpListener::bind("127.0.0.1", port) might succeed, but // lsof should detect the listener and the check should still fail. - let listener = match TcpListener::bind("[::]:0") { - Ok(l) => l, - Err(_) => return, // IPv6 not available, skip + let Ok(listener) = TcpListener::bind("[::]:0") else { + return; // IPv6 not available, skip }; let port = listener.local_addr().unwrap().port(); diff --git a/crates/openshell-core/src/inference.rs b/crates/openshell-core/src/inference.rs index a06c427f8b..0360cae5c4 100644 --- a/crates/openshell-core/src/inference.rs +++ b/crates/openshell-core/src/inference.rs @@ -28,7 +28,8 @@ pub enum AuthHeader { /// /// This is the single source of truth for provider-specific inference knowledge: /// default endpoint, supported protocols, credential key lookup order, auth -/// header style, and default headers. +/// header style, default headers, and allowed client-supplied passthrough +/// headers. /// /// This is separate from [`openshell_providers::ProviderPlugin`] which handles /// credential *discovery* (scanning env vars). `InferenceProviderProfile` handles @@ -45,6 +46,10 @@ pub struct InferenceProviderProfile { pub auth: AuthHeader, /// Default headers injected on every outgoing request. pub default_headers: &'static [(&'static str, &'static str)], + /// Client-supplied headers that may be forwarded to the upstream backend. + /// + /// Header names must be lowercase and must not include auth headers. + pub passthrough_headers: &'static [&'static str], } const OPENAI_PROTOCOLS: &[&str] = &[ @@ -64,6 +69,7 @@ static OPENAI_PROFILE: InferenceProviderProfile = InferenceProviderProfile { base_url_config_keys: &["OPENAI_BASE_URL"], auth: AuthHeader::Bearer, default_headers: &[], + passthrough_headers: &["openai-organization", "x-model-id"], }; static ANTHROPIC_PROFILE: InferenceProviderProfile = InferenceProviderProfile { @@ -74,6 +80,7 @@ static ANTHROPIC_PROFILE: InferenceProviderProfile = InferenceProviderProfile { base_url_config_keys: &["ANTHROPIC_BASE_URL"], auth: AuthHeader::Custom("x-api-key"), default_headers: &[("anthropic-version", "2023-06-01")], + passthrough_headers: &["anthropic-version", "anthropic-beta"], }; static NVIDIA_PROFILE: InferenceProviderProfile = InferenceProviderProfile { @@ -84,6 +91,7 @@ static NVIDIA_PROFILE: InferenceProviderProfile = InferenceProviderProfile { base_url_config_keys: &["NVIDIA_BASE_URL"], auth: AuthHeader::Bearer, default_headers: &[], + passthrough_headers: &["x-model-id"], }; /// Look up the inference provider profile for a given provider type. @@ -105,17 +113,33 @@ pub fn profile_for(provider_type: &str) -> Option<&'static InferenceProviderProf /// need the auth/header information (e.g. the sandbox bundle-to-route /// conversion). pub fn auth_for_provider_type(provider_type: &str) -> (AuthHeader, Vec<(String, String)>) { - match profile_for(provider_type) { - Some(profile) => { + let (auth, headers, _) = route_headers_for_provider_type(provider_type); + (auth, headers) +} + +/// Derive routing header policy for a provider type string. +/// +/// Returns the auth injection mode, route-level default headers, and the +/// allowed client-supplied passthrough headers for `inference.local`. +pub fn route_headers_for_provider_type( + provider_type: &str, +) -> (AuthHeader, Vec<(String, String)>, Vec) { + profile_for(provider_type).map_or_else( + || (AuthHeader::Bearer, Vec::new(), Vec::new()), + |profile| { let headers = profile .default_headers .iter() .map(|(k, v)| ((*k).to_string(), (*v).to_string())) .collect(); - (profile.auth.clone(), headers) - } - None => (AuthHeader::Bearer, Vec::new()), - } + let passthrough_headers = profile + .passthrough_headers + .iter() + .map(|name| (*name).to_string()) + .collect(); + (profile.auth.clone(), headers, passthrough_headers) + }, + ) } // --------------------------------------------------------------------------- @@ -193,6 +217,32 @@ mod tests { assert!(headers.iter().any(|(k, _)| k == "anthropic-version")); } + #[test] + fn route_headers_for_openai_include_passthrough_headers() { + let (_, _, passthrough_headers) = route_headers_for_provider_type("openai"); + assert!( + passthrough_headers + .iter() + .any(|name| name == "openai-organization") + ); + assert!(passthrough_headers.iter().any(|name| name == "x-model-id")); + } + + #[test] + fn route_headers_for_anthropic_include_passthrough_headers() { + let (_, _, passthrough_headers) = route_headers_for_provider_type("anthropic"); + assert!( + passthrough_headers + .iter() + .any(|name| name == "anthropic-version") + ); + assert!( + passthrough_headers + .iter() + .any(|name| name == "anthropic-beta") + ); + } + #[test] fn auth_for_openai_uses_bearer() { let (auth, headers) = auth_for_provider_type("openai"); @@ -206,4 +256,12 @@ mod tests { assert_eq!(auth, AuthHeader::Bearer); assert!(headers.is_empty()); } + + #[test] + fn route_headers_for_unknown_are_empty() { + let (auth, headers, passthrough_headers) = route_headers_for_provider_type("unknown"); + assert_eq!(auth, AuthHeader::Bearer); + assert!(headers.is_empty()); + assert!(passthrough_headers.is_empty()); + } } diff --git a/crates/openshell-core/src/lib.rs b/crates/openshell-core/src/lib.rs index c0b08f1a57..a4a1ea8225 100644 --- a/crates/openshell-core/src/lib.rs +++ b/crates/openshell-core/src/lib.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! OpenShell Core - shared library for OpenShell components. +//! `OpenShell` Core - shared library for `OpenShell` components. //! //! This crate provides: //! - Protocol buffer definitions and generated code @@ -14,13 +14,15 @@ pub mod error; pub mod forward; pub mod image; pub mod inference; +pub mod metadata; pub mod net; pub mod paths; pub mod proto; pub mod settings; -pub use config::{ComputeDriverKind, Config, TlsConfig}; -pub use error::{Error, Result}; +pub use config::{ComputeDriverKind, Config, OidcConfig, TlsConfig}; +pub use error::{ComputeDriverError, Error, Result}; +pub use metadata::{ObjectId, ObjectLabels, ObjectName}; /// Build version string derived from git metadata. /// diff --git a/crates/openshell-core/src/metadata.rs b/crates/openshell-core/src/metadata.rs new file mode 100644 index 0000000000..90566dcfd6 --- /dev/null +++ b/crates/openshell-core/src/metadata.rs @@ -0,0 +1,119 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Object metadata accessors for Kubernetes-style resources. +//! +//! These traits provide uniform access to `ObjectMeta` fields across all resource types. + +use crate::proto::{InferenceRoute, ObjectForTest, Provider, Sandbox, SshSession}; +use std::collections::HashMap; + +/// Provides access to the object's unique identifier. +pub trait ObjectId { + fn object_id(&self) -> &str; +} + +/// Provides access to the object's human-readable name. +pub trait ObjectName { + fn object_name(&self) -> &str; +} + +/// Provides access to the object's labels (key-value metadata). +pub trait ObjectLabels { + fn object_labels(&self) -> Option>; +} + +// Implementations for Sandbox +impl ObjectId for Sandbox { + fn object_id(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.id.as_str()) + } +} + +impl ObjectName for Sandbox { + fn object_name(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.name.as_str()) + } +} + +impl ObjectLabels for Sandbox { + fn object_labels(&self) -> Option> { + self.metadata.as_ref().map(|m| m.labels.clone()) + } +} + +// Implementations for Provider +impl ObjectId for Provider { + fn object_id(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.id.as_str()) + } +} + +impl ObjectName for Provider { + fn object_name(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.name.as_str()) + } +} + +impl ObjectLabels for Provider { + fn object_labels(&self) -> Option> { + self.metadata.as_ref().map(|m| m.labels.clone()) + } +} + +// Implementations for SshSession +impl ObjectId for SshSession { + fn object_id(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.id.as_str()) + } +} + +impl ObjectName for SshSession { + fn object_name(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.name.as_str()) + } +} + +impl ObjectLabels for SshSession { + fn object_labels(&self) -> Option> { + self.metadata.as_ref().map(|m| m.labels.clone()) + } +} + +// Implementations for InferenceRoute +impl ObjectId for InferenceRoute { + fn object_id(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.id.as_str()) + } +} + +impl ObjectName for InferenceRoute { + fn object_name(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.name.as_str()) + } +} + +impl ObjectLabels for InferenceRoute { + fn object_labels(&self) -> Option> { + self.metadata.as_ref().map(|m| m.labels.clone()) + } +} + +// Implementations for ObjectForTest (test-only proto type) +impl ObjectId for ObjectForTest { + fn object_id(&self) -> &str { + &self.id + } +} + +impl ObjectName for ObjectForTest { + fn object_name(&self) -> &str { + &self.name + } +} + +impl ObjectLabels for ObjectForTest { + fn object_labels(&self) -> Option> { + None + } +} diff --git a/crates/openshell-core/src/net.rs b/crates/openshell-core/src/net.rs index 3cd00d7b8a..5dca4feb67 100644 --- a/crates/openshell-core/src/net.rs +++ b/crates/openshell-core/src/net.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Network IP classification utilities shared across OpenShell crates. +//! Network IP classification utilities shared across `OpenShell` crates. //! //! These helpers enforce the always-blocked IP invariant (loopback, link-local, //! unspecified) and the broader internal-IP classification (adds RFC 1918 and @@ -97,11 +97,23 @@ pub fn is_always_blocked_net(net: ipnet::IpNet) -> bool { return true; } - // Check IPv4-mapped IPv6 (::ffff:127.0.0.1, ::ffff:169.254.x.x, etc.) - if let Some(v4) = network.to_ipv4_mapped() { - if v4.is_loopback() || v4.is_link_local() || v4.is_unspecified() { - return true; - } + // Check IPv4-mapped IPv6 addresses. The network-address check covers + // ranges whose first address is already in a blocked range + // (e.g. ::ffff:127.0.0.1/128, ::ffff:169.254.0.1/128). The + // containment checks below catch broader prefixes that only reach + // into a blocked range further in — e.g. ::ffff:168.0.0.0/103 + // has a public network address but spans ::ffff:169.254.0.0. + if network + .to_ipv4_mapped() + .is_some_and(|v4| v4.is_loopback() || v4.is_link_local() || v4.is_unspecified()) + { + return true; + } + if v6net.contains(&Ipv4Addr::LOCALHOST.to_ipv6_mapped()) + || v6net.contains(&Ipv4Addr::new(169, 254, 0, 0).to_ipv6_mapped()) + || v6net.contains(&Ipv4Addr::UNSPECIFIED.to_ipv6_mapped()) + { + return true; } false @@ -121,7 +133,7 @@ pub fn is_always_blocked_net(net: ipnet::IpNet) -> bool { /// when `allowed_ips` should be populated in proposals. pub fn is_internal_ip(ip: IpAddr) -> bool { match ip { - IpAddr::V4(v4) => is_internal_v4(&v4), + IpAddr::V4(v4) => is_internal_v4(v4), IpAddr::V6(v6) => { if v6.is_loopback() || v6.is_unspecified() { return true; @@ -136,7 +148,7 @@ pub fn is_internal_ip(ip: IpAddr) -> bool { } // Check IPv4-mapped IPv6 (::ffff:x.x.x.x) if let Some(v4) = v6.to_ipv4_mapped() { - return is_internal_v4(&v4); + return is_internal_v4(v4); } false } @@ -145,7 +157,7 @@ pub fn is_internal_ip(ip: IpAddr) -> bool { /// IPv4 internal address check covering RFC 1918, CGNAT (RFC 6598), and other /// special-use ranges that should never be reachable from sandbox egress. -fn is_internal_v4(v4: &Ipv4Addr) -> bool { +fn is_internal_v4(v4: Ipv4Addr) -> bool { if v4.is_loopback() || v4.is_private() || v4.is_link_local() || v4.is_unspecified() { return true; } @@ -332,6 +344,42 @@ mod tests { assert!(is_always_blocked_net(net)); } + #[test] + fn test_always_blocked_net_v6_ipv4_mapped_loopback_single() { + let net: ipnet::IpNet = "::ffff:127.0.0.1/128".parse().unwrap(); + assert!(is_always_blocked_net(net)); + } + + #[test] + fn test_always_blocked_net_v6_ipv4_mapped_link_local_single() { + let net: ipnet::IpNet = "::ffff:169.254.0.1/128".parse().unwrap(); + assert!(is_always_blocked_net(net)); + } + + #[test] + fn test_always_blocked_net_v6_ipv4_mapped_broad_spans_link_local() { + // ::ffff:168.0.0.0/103 has a public network address (168.0.0.0) but + // the range covers 168.0.0.0–169.255.255.255, which includes the + // link-local block 169.254.0.0/16. + let net: ipnet::IpNet = "::ffff:168.0.0.0/103".parse().unwrap(); + assert!(is_always_blocked_net(net)); + } + + #[test] + fn test_always_blocked_net_v6_ipv4_mapped_broad_spans_loopback() { + // ::ffff:64.0.0.0/98 has a public network address (64.0.0.0) but the + // range covers 64.0.0.0–127.255.255.255, which includes loopback. + let net: ipnet::IpNet = "::ffff:64.0.0.0/98".parse().unwrap(); + assert!(is_always_blocked_net(net)); + } + + #[test] + fn test_always_blocked_net_v6_ipv4_mapped_allows_public() { + // ::ffff:8.8.8.8/128 is a public address — should not be blocked. + let net: ipnet::IpNet = "::ffff:8.8.8.8/128".parse().unwrap(); + assert!(!is_always_blocked_net(net)); + } + // -- is_internal_ip -- #[test] diff --git a/crates/openshell-core/src/paths.rs b/crates/openshell-core/src/paths.rs index fd0a141b38..00104f3c2b 100644 --- a/crates/openshell-core/src/paths.rs +++ b/crates/openshell-core/src/paths.rs @@ -105,9 +105,7 @@ pub fn ensure_parent_dir_restricted(path: &Path) -> Result<()> { #[cfg(unix)] pub fn is_file_permissions_too_open(path: &Path) -> bool { use std::os::unix::fs::PermissionsExt; - std::fs::metadata(path) - .map(|m| m.permissions().mode() & 0o077 != 0) - .unwrap_or(false) + std::fs::metadata(path).is_ok_and(|m| m.permissions().mode() & 0o077 != 0) } #[cfg(test)] diff --git a/crates/openshell-core/src/settings.rs b/crates/openshell-core/src/settings.rs index 995fe6e2ae..2765ebeda3 100644 --- a/crates/openshell-core/src/settings.rs +++ b/crates/openshell-core/src/settings.rs @@ -48,7 +48,15 @@ pub struct RegisteredSetting { /// settable via `settings set`. The server validates that only registered /// keys are accepted. /// 5. Add a unit test in this module's `tests` section to cover the new key. +pub const PROVIDERS_V2_ENABLED_KEY: &str = "providers_v2_enabled"; + pub const REGISTERED_SETTINGS: &[RegisteredSetting] = &[ + // Gateway-level opt-in for provider profile policy composition. Defaults + // to false when unset. + RegisteredSetting { + key: PROVIDERS_V2_ENABLED_KEY, + kind: SettingValueKind::Bool, + }, // When true the sandbox writes OCSF v1.7.0 JSONL records to // `/var/log/openshell-ocsf*.log` (daily rotation, 3 files) in addition // to the human-readable shorthand log. Defaults to false (no JSONL written). @@ -99,8 +107,8 @@ pub fn parse_bool_like(raw: &str) -> Option { #[cfg(test)] mod tests { use super::{ - REGISTERED_SETTINGS, RegisteredSetting, SettingValueKind, parse_bool_like, - registered_keys_csv, setting_for_key, + PROVIDERS_V2_ENABLED_KEY, REGISTERED_SETTINGS, RegisteredSetting, SettingValueKind, + parse_bool_like, registered_keys_csv, setting_for_key, }; #[cfg(feature = "dev-settings")] @@ -123,6 +131,13 @@ mod tests { assert!(setting_for_key("policy").is_none()); } + #[test] + fn setting_for_key_returns_providers_v2_enabled() { + let setting = setting_for_key(PROVIDERS_V2_ENABLED_KEY) + .expect("providers_v2_enabled should be registered"); + assert_eq!(setting.kind, SettingValueKind::Bool); + } + // ---- parse_bool_like ---- #[test] diff --git a/crates/openshell-driver-docker/Cargo.toml b/crates/openshell-driver-docker/Cargo.toml new file mode 100644 index 0000000000..79d4fb37d5 --- /dev/null +++ b/crates/openshell-driver-docker/Cargo.toml @@ -0,0 +1,28 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-driver-docker" +description = "Docker compute driver for OpenShell" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +openshell-core = { path = "../openshell-core" } + +tokio = { workspace = true } +tonic = { workspace = true } +futures = { workspace = true } +tokio-stream = { workspace = true } +tracing = { workspace = true } +bytes = { workspace = true } +bollard = { version = "0.20" } +tar = "0.4" +tempfile = "3" +url = { workspace = true } + +[lints] +workspace = true diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs new file mode 100644 index 0000000000..0eaef3bceb --- /dev/null +++ b/crates/openshell-driver-docker/src/lib.rs @@ -0,0 +1,2049 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Docker compute driver. + +#![allow(clippy::result_large_err)] + +use bollard::Docker; +use bollard::errors::Error as BollardError; +use bollard::models::{ + ContainerCreateBody, ContainerSummary, ContainerSummaryStateEnum, DeviceRequest, + EndpointSettings, HostConfig, Mount, MountTypeEnum, NetworkCreateRequest, NetworkingConfig, + RestartPolicy, RestartPolicyNameEnum, SystemInfo, +}; +use bollard::query_parameters::{ + CreateContainerOptionsBuilder, CreateImageOptions, DownloadFromContainerOptionsBuilder, + ListContainersOptionsBuilder, RemoveContainerOptionsBuilder, StopContainerOptionsBuilder, +}; +use bytes::Bytes; +use futures::{Stream, StreamExt}; +use openshell_core::config::{ + CDI_GPU_DEVICE_ALL, DEFAULT_DOCKER_NETWORK_NAME, DEFAULT_STOP_TIMEOUT_SECS, +}; +use openshell_core::proto::compute::v1::{ + CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, + DriverCondition, DriverSandbox, DriverSandboxStatus, DriverSandboxTemplate, + GetCapabilitiesRequest, GetCapabilitiesResponse, GetSandboxRequest, GetSandboxResponse, + ListSandboxesRequest, ListSandboxesResponse, StopSandboxRequest, StopSandboxResponse, + ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesDeletedEvent, + WatchSandboxesEvent, WatchSandboxesRequest, WatchSandboxesSandboxEvent, + compute_driver_server::ComputeDriver, watch_sandboxes_event, +}; +use openshell_core::{Config, Error, Result as CoreResult}; +use std::collections::HashMap; +use std::io::Read; +use std::net::{IpAddr, SocketAddr}; +use std::path::{Path, PathBuf}; +use std::pin::Pin; +use std::sync::Arc; +use std::time::Duration; +use tokio::sync::{broadcast, mpsc}; +use tokio_stream::wrappers::ReceiverStream; +use tonic::{Request, Response, Status}; +use tracing::{info, warn}; +use url::Url; + +const WATCH_BUFFER: usize = 128; +const WATCH_POLL_INTERVAL: Duration = Duration::from_secs(2); +const WATCH_POLL_MAX_BACKOFF: Duration = Duration::from_secs(30); + +const MANAGED_BY_LABEL_KEY: &str = "openshell.ai/managed-by"; +const MANAGED_BY_LABEL_VALUE: &str = "openshell"; +const SANDBOX_ID_LABEL_KEY: &str = "openshell.ai/sandbox-id"; +const SANDBOX_NAME_LABEL_KEY: &str = "openshell.ai/sandbox-name"; +const SANDBOX_NAMESPACE_LABEL_KEY: &str = "openshell.ai/sandbox-namespace"; + +const SUPERVISOR_MOUNT_PATH: &str = "/opt/openshell/bin/openshell-sandbox"; +const TLS_CA_MOUNT_PATH: &str = "/etc/openshell/tls/client/ca.crt"; +const TLS_CERT_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.crt"; +const TLS_KEY_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.key"; +const SANDBOX_COMMAND: &str = "sleep infinity"; +const SUPERVISOR_PATH: &str = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"; +const HOST_OPENSHELL_INTERNAL: &str = "host.openshell.internal"; +const HOST_DOCKER_INTERNAL: &str = "host.docker.internal"; +const DOCKER_NETWORK_DRIVER: &str = "bridge"; + +/// Default image holding the Linux `openshell-sandbox` binary. The gateway +/// pulls this image and extracts the binary to a host-side cache when no +/// explicit `--docker-supervisor-bin` override or local build is available. +const DEFAULT_DOCKER_SUPERVISOR_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/supervisor"; + +/// Path to the supervisor binary inside the `openshell/supervisor` image +/// (a `FROM scratch` image containing only the binary). +const SUPERVISOR_IMAGE_BINARY_PATH: &str = "/openshell-sandbox"; + +/// Return the default `ghcr.io/nvidia/openshell/supervisor:` reference +/// used when no supervisor binary override is provided. +pub fn default_docker_supervisor_image() -> String { + format!( + "{DEFAULT_DOCKER_SUPERVISOR_IMAGE_REPO}:{}", + default_docker_supervisor_image_tag() + ) +} + +/// Image tag baked in at compile time to pair the gateway with a matching +/// supervisor image. +/// +/// Build pipelines pass `OPENSHELL_IMAGE_TAG` explicitly. The `IMAGE_TAG` +/// fallback covers image build wrappers that already tag the gateway and +/// supervisor together. Standalone release binaries also patch the Cargo +/// package version, so use it when it has been set to a real release value. +fn default_docker_supervisor_image_tag() -> &'static str { + resolve_default_docker_supervisor_image_tag( + option_env!("OPENSHELL_IMAGE_TAG"), + option_env!("IMAGE_TAG"), + env!("CARGO_PKG_VERSION"), + ) +} + +fn resolve_default_docker_supervisor_image_tag( + openshell_image_tag: Option<&'static str>, + image_tag: Option<&'static str>, + cargo_pkg_version: &'static str, +) -> &'static str { + openshell_image_tag + .filter(|tag| !tag.is_empty()) + .or_else(|| image_tag.filter(|tag| !tag.is_empty())) + .unwrap_or_else(|| { + if cargo_pkg_version.is_empty() || cargo_pkg_version == "0.0.0" { + "dev" + } else { + cargo_pkg_version + } + }) +} + +/// Queried by the Docker driver to decide when a sandbox's supervisor +/// relay is live. Implementations return `true` once a sandbox has an +/// active `ConnectSupervisor` session registered. +/// +/// The driver cannot observe the supervisor's SSH socket directly (it +/// lives inside the container), so it leans on this signal to flip the +/// Ready condition from `DependenciesNotReady` to `True`. +pub trait SupervisorReadiness: Send + Sync + 'static { + fn is_supervisor_connected(&self, sandbox_id: &str) -> bool; +} + +/// Gateway-local configuration for the Docker compute driver. +#[derive(Debug, Clone, Default)] +pub struct DockerComputeConfig { + /// Optional override for the Linux `openshell-sandbox` binary mounted into containers. + pub supervisor_bin: Option, + + /// Optional override for the image the gateway pulls to extract the + /// Linux `openshell-sandbox` binary when no explicit binary path or + /// local build is available. Defaults to + /// `ghcr.io/nvidia/openshell/supervisor:`. + pub supervisor_image: Option, + + /// Host-side CA certificate for Docker sandbox mTLS. + pub guest_tls_ca: Option, + + /// Host-side client certificate for Docker sandbox mTLS. + pub guest_tls_cert: Option, + + /// Host-side private key for Docker sandbox mTLS. + pub guest_tls_key: Option, + + /// Docker bridge network that sandbox containers join. + pub network_name: String, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct DockerGuestTlsPaths { + pub(crate) ca: PathBuf, + pub(crate) cert: PathBuf, + pub(crate) key: PathBuf, +} + +#[derive(Debug, Clone)] +struct DockerDriverRuntimeConfig { + default_image: String, + image_pull_policy: String, + sandbox_namespace: String, + grpc_endpoint: String, + network_name: String, + gateway_route: DockerGatewayRoute, + ssh_socket_path: String, + stop_timeout_secs: u32, + log_level: String, + supervisor_bin: PathBuf, + guest_tls: Option, + daemon_version: String, + supports_gpu: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum DockerGatewayRoute { + Bridge { + bind_address: SocketAddr, + host_alias_ip: IpAddr, + }, + HostGateway, +} + +#[derive(Clone)] +pub struct DockerComputeDriver { + docker: Arc, + config: DockerDriverRuntimeConfig, + events: broadcast::Sender, + supervisor_readiness: Arc, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +struct DockerResourceLimits { + nano_cpus: Option, + memory_bytes: Option, +} + +type WatchStream = + Pin> + Send + 'static>>; + +impl DockerComputeDriver { + pub async fn new( + config: &Config, + docker_config: &DockerComputeConfig, + supervisor_readiness: Arc, + ) -> CoreResult { + if config.grpc_endpoint.trim().is_empty() { + return Err(Error::config( + "grpc_endpoint is required when using the docker compute driver", + )); + } + + let docker = Docker::connect_with_local_defaults() + .map_err(|err| Error::execution(format!("failed to create Docker client: {err}")))?; + let version = docker.version().await.map_err(|err| { + Error::execution(format!("failed to query Docker daemon version: {err}")) + })?; + let info = docker.info().await.map_err(|err| { + Error::execution(format!("failed to query Docker daemon info: {err}")) + })?; + let supports_gpu = info + .cdi_spec_dirs + .as_ref() + .is_some_and(|dirs| !dirs.is_empty()); + let gateway_port = config.bind_address.port(); + if gateway_port == 0 { + return Err(Error::config( + "docker compute driver requires a fixed non-zero gateway bind port", + )); + } + let network_name = docker_network_name(docker_config); + let bridge_gateway_ip = ensure_bridge_network(&docker, &network_name).await?; + let host_gateway_ip = parse_optional_host_gateway_ip(&config.host_gateway_ip)?; + let gateway_route = + docker_gateway_route(&info, bridge_gateway_ip, gateway_port, host_gateway_ip); + let grpc_endpoint = docker_container_openshell_endpoint( + &config.grpc_endpoint, + HOST_OPENSHELL_INTERNAL, + gateway_port, + ); + let daemon_arch = normalize_docker_arch(version.arch.as_deref().unwrap_or_default()); + let supervisor_bin = resolve_supervisor_bin(&docker, docker_config, &daemon_arch).await?; + let guest_tls = docker_guest_tls_paths(config, docker_config)?; + + let driver = Self { + docker: Arc::new(docker), + config: DockerDriverRuntimeConfig { + default_image: config.sandbox_image.clone(), + image_pull_policy: config.sandbox_image_pull_policy.clone(), + sandbox_namespace: config.sandbox_namespace.clone(), + grpc_endpoint, + network_name, + gateway_route, + ssh_socket_path: config.sandbox_ssh_socket_path.clone(), + stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS, + log_level: config.log_level.clone(), + supervisor_bin, + guest_tls, + daemon_version: version.version.unwrap_or_else(|| "unknown".to_string()), + supports_gpu, + }, + events: broadcast::channel(WATCH_BUFFER).0, + supervisor_readiness, + }; + + let poll_driver = driver.clone(); + tokio::spawn(async move { + poll_driver.poll_loop().await; + }); + + Ok(driver) + } + + #[must_use] + pub fn gateway_bind_addresses(&self) -> Vec { + match self.config.gateway_route { + DockerGatewayRoute::Bridge { bind_address, .. } => vec![bind_address], + DockerGatewayRoute::HostGateway => Vec::new(), + } + } + + fn capabilities(&self) -> GetCapabilitiesResponse { + GetCapabilitiesResponse { + driver_name: "docker".to_string(), + driver_version: self.config.daemon_version.clone(), + default_image: self.config.default_image.clone(), + supports_gpu: self.config.supports_gpu, + gpu_count: 0, + } + } + + fn validate_sandbox( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + ) -> Result<(), Status> { + let spec = sandbox + .spec + .as_ref() + .ok_or_else(|| Status::invalid_argument("sandbox.spec is required"))?; + let template = spec + .template + .as_ref() + .ok_or_else(|| Status::invalid_argument("sandbox.spec.template is required"))?; + + if template.image.trim().is_empty() { + return Err(Status::failed_precondition( + "docker sandboxes require a template image", + )); + } + if spec.gpu && !config.supports_gpu { + return Err(Status::failed_precondition( + "docker GPU sandboxes require Docker CDI support. Enable CDI on the Docker daemon, then restart the OpenShell gateway/server so GPU capability is detected.", + )); + } + if !template.agent_socket_path.trim().is_empty() { + return Err(Status::failed_precondition( + "docker compute driver does not support template.agent_socket_path", + )); + } + if template + .platform_config + .as_ref() + .is_some_and(|config| !config.fields.is_empty()) + { + return Err(Status::failed_precondition( + "docker compute driver does not support template.platform_config", + )); + } + + let _ = docker_resource_limits(template)?; + Ok(()) + } + + async fn get_sandbox_snapshot( + &self, + sandbox_id: &str, + sandbox_name: &str, + ) -> Result, Status> { + let container = self + .find_managed_container_summary(sandbox_id, sandbox_name) + .await?; + Ok(container.and_then(|summary| { + sandbox_from_container_summary(&summary, self.supervisor_readiness.as_ref()) + })) + } + + async fn current_snapshots(&self) -> Result, Status> { + let containers = self.list_managed_container_summaries().await?; + let mut sandboxes = containers + .iter() + .filter_map(|summary| { + sandbox_from_container_summary(summary, self.supervisor_readiness.as_ref()) + }) + .collect::>(); + sandboxes.sort_by(|left, right| left.id.cmp(&right.id)); + Ok(sandboxes) + } + + async fn create_sandbox_inner(&self, sandbox: &DriverSandbox) -> Result<(), Status> { + Self::validate_sandbox(sandbox, &self.config)?; + + if self + .find_managed_container_summary(&sandbox.id, &sandbox.name) + .await? + .is_some() + { + return Err(Status::already_exists("sandbox already exists")); + } + + let template = sandbox + .spec + .as_ref() + .and_then(|spec| spec.template.as_ref()) + .expect("validated sandbox has template"); + self.ensure_image_available(&template.image).await?; + + let container_name = container_name_for_sandbox(sandbox); + let create_body = build_container_create_body(sandbox, &self.config)?; + self.docker + .create_container( + Some( + CreateContainerOptionsBuilder::default() + .name(container_name.as_str()) + .build(), + ), + create_body, + ) + .await + .map_err(|err| { + create_status_from_docker_error("create docker sandbox container", err) + })?; + + if let Err(err) = self.docker.start_container(&container_name, None).await { + let cleanup = self + .docker + .remove_container( + &container_name, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + if let Err(cleanup_err) = cleanup { + warn!( + sandbox_id = %sandbox.id, + container_name, + error = %cleanup_err, + "Failed to clean up Docker container after start failure" + ); + } + return Err(create_status_from_docker_error( + "start docker sandbox container", + err, + )); + } + + Ok(()) + } + + async fn delete_sandbox_inner( + &self, + sandbox_id: &str, + sandbox_name: &str, + ) -> Result { + let Some(container) = self + .find_managed_container_summary(sandbox_id, sandbox_name) + .await? + else { + return Ok(false); + }; + let Some(target) = summary_container_target(&container) else { + return Ok(false); + }; + + match self + .docker + .remove_container( + &target, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await + { + Ok(()) => Ok(true), + Err(err) if is_not_found_error(&err) => Ok(false), + Err(err) => Err(internal_status("delete docker sandbox container", err)), + } + } + + async fn stop_sandbox_inner(&self, sandbox_id: &str, sandbox_name: &str) -> Result<(), Status> { + let Some(container) = self + .find_managed_container_summary(sandbox_id, sandbox_name) + .await? + else { + return Err(Status::not_found("sandbox not found")); + }; + let Some(target) = summary_container_target(&container) else { + return Err(Status::not_found("sandbox container has no id or name")); + }; + + match self + .docker + .stop_container( + &target, + Some( + StopContainerOptionsBuilder::default() + .t(docker_stop_timeout_secs(self.config.stop_timeout_secs)) + .build(), + ), + ) + .await + { + Ok(()) => Ok(()), + Err(err) if is_not_modified_error(&err) => Ok(()), + Err(err) if is_not_found_error(&err) => Err(Status::not_found("sandbox not found")), + Err(err) => Err(internal_status("stop docker sandbox container", err)), + } + } + + /// Start a managed sandbox container that was previously stopped. Used + /// by the gateway to resume sandboxes after a restart so that running + /// state in the gateway store is matched by an actually-running + /// container. + /// + /// Returns `Ok(true)` when a container existed and was started (or was + /// already running), `Ok(false)` when no managed container is found for + /// the sandbox, and `Err(...)` for any Docker failure. + pub async fn resume_sandbox( + &self, + sandbox_id: &str, + sandbox_name: &str, + ) -> Result { + let Some(container) = self + .find_managed_container_summary(sandbox_id, sandbox_name) + .await? + else { + return Ok(false); + }; + let Some(target) = summary_container_target(&container) else { + return Ok(false); + }; + let state = container.state.unwrap_or(ContainerSummaryStateEnum::EMPTY); + if !container_state_needs_resume(state) { + return Ok(true); + } + + match self.docker.start_container(&target, None).await { + Ok(()) => Ok(true), + // Already running — race with another resume path or the + // restart policy. Treat as success. + Err(err) if is_not_modified_error(&err) => Ok(true), + Err(err) if is_not_found_error(&err) => Ok(false), + Err(err) => Err(internal_status("start docker sandbox container", err)), + } + } + + pub async fn stop_managed_containers_on_shutdown(&self) -> Result { + let containers = self.list_managed_container_summaries().await?; + let targets = containers + .into_iter() + .filter_map(|container| { + let state = container.state.unwrap_or(ContainerSummaryStateEnum::EMPTY); + if container_state_needs_shutdown_stop(state) { + summary_container_target(&container) + } else { + None + } + }) + .collect::>(); + let target_count = targets.len(); + let mut stopped = 0usize; + let mut failures = Vec::new(); + let stop_timeout_secs = self.config.stop_timeout_secs; + + let mut stop_results = futures::stream::iter(targets.into_iter().map(|target| { + let docker = self.docker.clone(); + async move { + let result = docker + .stop_container( + &target, + Some( + StopContainerOptionsBuilder::default() + .t(docker_stop_timeout_secs(stop_timeout_secs)) + .build(), + ), + ) + .await; + (target, result) + } + })) + .buffer_unordered(16); + + while let Some((target, result)) = stop_results.next().await { + match result { + Ok(()) => { + stopped += 1; + } + Err(err) if is_not_found_error(&err) || is_not_modified_error(&err) => {} + Err(err) => { + warn!( + container = %target, + error = %err, + "Failed to stop Docker sandbox container during shutdown" + ); + failures.push(target); + } + } + } + + if !failures.is_empty() { + return Err(Status::internal(format!( + "failed to stop {} of {target_count} Docker sandbox containers during shutdown", + failures.len() + ))); + } + + Ok(stopped) + } + + async fn poll_loop(self) { + let mut previous = match self.current_snapshot_map().await { + Ok(snapshots) => snapshots, + Err(err) => { + warn!(error = %err, "Failed to seed Docker sandbox watch state"); + HashMap::new() + } + }; + + // Exponential backoff on consecutive Docker failures to avoid a 2s + // warn-log flood when the daemon is unreachable for an extended + // period (e.g. restart, socket removed). + let mut backoff = WATCH_POLL_INTERVAL; + loop { + tokio::time::sleep(backoff).await; + match self.current_snapshot_map().await { + Ok(current) => { + emit_snapshot_diff(&self.events, &previous, ¤t); + previous = current; + backoff = WATCH_POLL_INTERVAL; + } + Err(err) => { + warn!( + error = %err, + backoff_secs = backoff.as_secs(), + "Failed to poll Docker sandboxes" + ); + backoff = (backoff * 2).min(WATCH_POLL_MAX_BACKOFF); + } + } + } + } + + async fn current_snapshot_map(&self) -> Result, Status> { + self.current_snapshots().await.map(|snapshots| { + snapshots + .into_iter() + .map(|sandbox| (sandbox.id.clone(), sandbox)) + .collect() + }) + } + + async fn list_managed_container_summaries(&self) -> Result, Status> { + let filters = managed_container_label_filters(&self.config.sandbox_namespace, []); + self.docker + .list_containers(Some( + ListContainersOptionsBuilder::default() + .all(true) + .filters(&filters) + .build(), + )) + .await + .map_err(|err| internal_status("list Docker sandbox containers", err)) + } + + async fn find_managed_container_summary( + &self, + sandbox_id: &str, + sandbox_name: &str, + ) -> Result, Status> { + let mut label_filter_values = Vec::new(); + if !sandbox_id.is_empty() { + label_filter_values.push(format!("{SANDBOX_ID_LABEL_KEY}={sandbox_id}")); + } else if !sandbox_name.is_empty() { + label_filter_values.push(format!("{SANDBOX_NAME_LABEL_KEY}={sandbox_name}")); + } + + let filters = + managed_container_label_filters(&self.config.sandbox_namespace, label_filter_values); + let containers = self + .docker + .list_containers(Some( + ListContainersOptionsBuilder::default() + .all(true) + .filters(&filters) + .build(), + )) + .await + .map_err(|err| internal_status("find Docker sandbox container", err))?; + + Ok(containers.into_iter().find(|summary| { + let Some(labels) = summary.labels.as_ref() else { + return false; + }; + let namespace_matches = labels + .get(SANDBOX_NAMESPACE_LABEL_KEY) + .is_some_and(|value| value == &self.config.sandbox_namespace); + let id_matches = sandbox_id.is_empty() + || labels + .get(SANDBOX_ID_LABEL_KEY) + .is_some_and(|value| value == sandbox_id); + let name_matches = sandbox_name.is_empty() + || labels + .get(SANDBOX_NAME_LABEL_KEY) + .is_some_and(|value| value == sandbox_name); + namespace_matches && id_matches && name_matches + })) + } + + async fn ensure_image_available(&self, image: &str) -> Result<(), Status> { + let policy = self.config.image_pull_policy.trim().to_ascii_lowercase(); + match policy.as_str() { + "" | "ifnotpresent" => { + if self.docker.inspect_image(image).await.is_ok() { + return Ok(()); + } + self.pull_image(image).await + } + "always" => self.pull_image(image).await, + "never" => match self.docker.inspect_image(image).await { + Ok(_) => Ok(()), + Err(err) if is_not_found_error(&err) => Err(Status::failed_precondition(format!( + "docker image '{image}' is not present locally and sandbox_image_pull_policy=Never" + ))), + Err(err) => Err(internal_status("inspect Docker image", err)), + }, + other => Err(Status::failed_precondition(format!( + "unsupported docker sandbox_image_pull_policy '{other}'; expected Always, IfNotPresent, or Never", + ))), + } + } + + async fn pull_image(&self, image: &str) -> Result<(), Status> { + let mut stream = self.docker.create_image( + Some(CreateImageOptions { + from_image: Some(image.to_string()), + ..Default::default() + }), + None, + None, + ); + while let Some(result) = stream.next().await { + result.map_err(|err| internal_status("pull Docker image", err))?; + } + Ok(()) + } +} + +#[tonic::async_trait] +impl ComputeDriver for DockerComputeDriver { + type WatchSandboxesStream = WatchStream; + + async fn get_capabilities( + &self, + _request: Request, + ) -> Result, Status> { + Ok(Response::new(self.capabilities())) + } + + async fn validate_sandbox_create( + &self, + request: Request, + ) -> Result, Status> { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + Self::validate_sandbox(&sandbox, &self.config)?; + Ok(Response::new(ValidateSandboxCreateResponse {})) + } + + async fn get_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + require_sandbox_identifier(&request.sandbox_id, &request.sandbox_name)?; + + let sandbox = self + .get_sandbox_snapshot(&request.sandbox_id, &request.sandbox_name) + .await? + .ok_or_else(|| Status::not_found("sandbox not found"))?; + + if !request.sandbox_id.is_empty() && request.sandbox_id != sandbox.id { + return Err(Status::failed_precondition( + "sandbox_id did not match the fetched sandbox", + )); + } + + Ok(Response::new(GetSandboxResponse { + sandbox: Some(sandbox), + })) + } + + async fn list_sandboxes( + &self, + _request: Request, + ) -> Result, Status> { + Ok(Response::new(ListSandboxesResponse { + sandboxes: self.current_snapshots().await?, + })) + } + + async fn create_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + self.create_sandbox_inner(&sandbox).await?; + Ok(Response::new(CreateSandboxResponse {})) + } + + async fn stop_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + require_sandbox_identifier(&request.sandbox_id, &request.sandbox_name)?; + + self.stop_sandbox_inner(&request.sandbox_id, &request.sandbox_name) + .await?; + Ok(Response::new(StopSandboxResponse {})) + } + + async fn delete_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + require_sandbox_identifier(&request.sandbox_id, &request.sandbox_name)?; + + let event_sandbox_id = request.sandbox_id.clone(); + let deleted = self + .delete_sandbox_inner(&request.sandbox_id, &request.sandbox_name) + .await?; + if deleted && !event_sandbox_id.is_empty() { + let _ = self.events.send(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Deleted( + WatchSandboxesDeletedEvent { + sandbox_id: event_sandbox_id, + }, + )), + }); + } + + Ok(Response::new(DeleteSandboxResponse { deleted })) + } + + async fn watch_sandboxes( + &self, + _request: Request, + ) -> Result, Status> { + // Subscribe before taking the initial snapshot so any event emitted + // between the snapshot and this subscriber becoming active is still + // delivered. Downstream consumers treat sandbox events as + // idempotent (keyed by sandbox id), so a duplicate event is benign + // while a missed one leaks state. + let mut rx = self.events.subscribe(); + let initial = self.current_snapshots().await?; + let (tx, out_rx) = mpsc::channel(WATCH_BUFFER); + tokio::spawn(async move { + for sandbox in initial { + if tx + .send(Ok(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { + sandbox: Some(sandbox), + }, + )), + })) + .await + .is_err() + { + return; + } + } + + loop { + match rx.recv().await { + Ok(event) => { + if tx.send(Ok(event)).await.is_err() { + return; + } + } + Err(broadcast::error::RecvError::Lagged(_)) => {} + Err(broadcast::error::RecvError::Closed) => return, + } + } + }); + + Ok(Response::new(Box::pin(ReceiverStream::new(out_rx)))) + } +} + +fn build_mounts(config: &DockerDriverRuntimeConfig) -> Vec { + let mut mounts = vec![bind_mount( + &config.supervisor_bin, + SUPERVISOR_MOUNT_PATH, + true, + )]; + if let Some(tls) = &config.guest_tls { + mounts.push(bind_mount(&tls.ca, TLS_CA_MOUNT_PATH, true)); + mounts.push(bind_mount(&tls.cert, TLS_CERT_MOUNT_PATH, true)); + mounts.push(bind_mount(&tls.key, TLS_KEY_MOUNT_PATH, true)); + } + mounts +} + +fn bind_mount(source: &Path, target: &str, read_only: bool) -> Mount { + Mount { + target: Some(target.to_string()), + source: Some(source.display().to_string()), + typ: Some(MountTypeEnum::BIND), + read_only: Some(read_only), + ..Default::default() + } +} + +fn build_environment(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) -> Vec { + let mut environment = HashMap::from([ + ("HOME".to_string(), "/root".to_string()), + ("PATH".to_string(), SUPERVISOR_PATH.to_string()), + ("TERM".to_string(), "xterm".to_string()), + ( + "OPENSHELL_LOG_LEVEL".to_string(), + sandbox_log_level(sandbox, &config.log_level), + ), + ]); + + if let Some(spec) = sandbox.spec.as_ref() { + if let Some(template) = spec.template.as_ref() { + environment.extend(template.environment.clone()); + } + environment.extend(spec.environment.clone()); + } + + environment.insert( + "OPENSHELL_ENDPOINT".to_string(), + config.grpc_endpoint.clone(), + ); + environment.insert("OPENSHELL_SANDBOX_ID".to_string(), sandbox.id.clone()); + environment.insert("OPENSHELL_SANDBOX".to_string(), sandbox.name.clone()); + environment.insert( + "OPENSHELL_SSH_SOCKET_PATH".to_string(), + config.ssh_socket_path.clone(), + ); + environment.insert( + "OPENSHELL_SANDBOX_COMMAND".to_string(), + SANDBOX_COMMAND.to_string(), + ); + // The root supervisor executes namespace helpers during bootstrap; keep + // their search path driver-owned even when the template/spec set PATH. + environment.insert("PATH".to_string(), SUPERVISOR_PATH.to_string()); + if config.guest_tls.is_some() { + environment.insert( + "OPENSHELL_TLS_CA".to_string(), + TLS_CA_MOUNT_PATH.to_string(), + ); + environment.insert( + "OPENSHELL_TLS_CERT".to_string(), + TLS_CERT_MOUNT_PATH.to_string(), + ); + environment.insert( + "OPENSHELL_TLS_KEY".to_string(), + TLS_KEY_MOUNT_PATH.to_string(), + ); + } + + let mut pairs = environment.into_iter().collect::>(); + pairs.sort_by(|left, right| left.0.cmp(&right.0)); + pairs + .into_iter() + .map(|(key, value)| format!("{key}={value}")) + .collect() +} + +fn docker_gpu_device_requests(gpu: bool) -> Option> { + gpu.then(|| { + vec![DeviceRequest { + driver: Some("cdi".to_string()), + device_ids: Some(vec![CDI_GPU_DEVICE_ALL.to_string()]), + ..Default::default() + }] + }) +} + +fn build_container_create_body( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, +) -> Result { + let spec = sandbox + .spec + .as_ref() + .ok_or_else(|| Status::invalid_argument("sandbox.spec is required"))?; + let template = spec + .template + .as_ref() + .ok_or_else(|| Status::invalid_argument("sandbox.spec.template is required"))?; + let resource_limits = docker_resource_limits(template)?; + let mut labels = template.labels.clone(); + labels.insert( + MANAGED_BY_LABEL_KEY.to_string(), + MANAGED_BY_LABEL_VALUE.to_string(), + ); + labels.insert(SANDBOX_ID_LABEL_KEY.to_string(), sandbox.id.clone()); + labels.insert(SANDBOX_NAME_LABEL_KEY.to_string(), sandbox.name.clone()); + // The list/get/find paths filter by `config.sandbox_namespace`, so use + // the same value here. `DriverSandbox.namespace` is unset on the request + // path (the gateway elides it), and using it would produce containers + // that the driver itself cannot find afterwards. + labels.insert( + SANDBOX_NAMESPACE_LABEL_KEY.to_string(), + config.sandbox_namespace.clone(), + ); + + Ok(ContainerCreateBody { + image: Some(template.image.clone()), + user: Some("0".to_string()), + env: Some(build_environment(sandbox, config)), + entrypoint: Some(vec![SUPERVISOR_MOUNT_PATH.to_string()]), + // Clear the image CMD so Docker does not append inherited args to the + // supervisor entrypoint. + cmd: Some(Vec::new()), + labels: Some(labels), + host_config: Some(HostConfig { + nano_cpus: resource_limits.nano_cpus, + memory: resource_limits.memory_bytes, + device_requests: docker_gpu_device_requests(spec.gpu), + mounts: Some(build_mounts(config)), + restart_policy: Some(RestartPolicy { + name: Some(RestartPolicyNameEnum::UNLESS_STOPPED), + maximum_retry_count: None, + }), + cap_add: Some(vec![ + "SYS_ADMIN".to_string(), + "NET_ADMIN".to_string(), + "SYS_PTRACE".to_string(), + "SYSLOG".to_string(), + ]), + // The sandbox supervisor needs to bind-mount `/run/netns`, + // mark it shared, and create per-process network namespaces. + // Docker's default AppArmor profile (`docker-default`) denies + // these mount operations even with CAP_SYS_ADMIN, so we opt + // out of AppArmor confinement for sandbox containers. The + // sandbox enforces its own security boundary via Landlock, + // seccomp, OPA policy evaluation, and the dedicated network + // namespace it sets up for the agent — AppArmor at the + // container layer is redundant relative to those controls + // and conflicts with them in this case. + security_opt: Some(vec!["apparmor=unconfined".to_string()]), + network_mode: Some(config.network_name.clone()), + extra_hosts: Some(docker_extra_hosts(&config.gateway_route)), + ..Default::default() + }), + networking_config: Some(NetworkingConfig { + endpoints_config: Some(HashMap::from([( + config.network_name.clone(), + EndpointSettings::default(), + )])), + }), + ..Default::default() + }) +} + +/// Reject driver requests that arrive with neither a sandbox id nor a +/// sandbox name. Without this guard, downstream label filters degenerate +/// to "match every managed container in the namespace", which would let +/// `delete_sandbox`/`stop_sandbox`/`get_sandbox` pick an arbitrary +/// sandbox out of the set the driver manages. +fn require_sandbox_identifier(sandbox_id: &str, sandbox_name: &str) -> Result<(), Status> { + if sandbox_id.is_empty() && sandbox_name.is_empty() { + return Err(Status::invalid_argument( + "sandbox_id or sandbox_name is required", + )); + } + Ok(()) +} + +fn sandbox_log_level(sandbox: &DriverSandbox, default_level: &str) -> String { + sandbox + .spec + .as_ref() + .map(|spec| spec.log_level.as_str()) + .filter(|level| !level.is_empty()) + .unwrap_or(default_level) + .to_string() +} + +fn docker_container_openshell_endpoint(endpoint: &str, host: &str, port: u16) -> String { + let Ok(mut url) = Url::parse(endpoint) else { + return endpoint.to_string(); + }; + + if url.set_host(Some(host)).is_ok() && url.set_port(Some(port)).is_ok() { + return url.to_string(); + } + + endpoint.to_string() +} + +fn docker_network_name(config: &DockerComputeConfig) -> String { + let name = config.network_name.trim(); + if name.is_empty() { + return DEFAULT_DOCKER_NETWORK_NAME.to_string(); + } + name.to_string() +} + +fn parse_optional_host_gateway_ip(value: &str) -> CoreResult> { + let trimmed = value.trim(); + if trimmed.is_empty() { + return Ok(None); + } + + trimmed.parse().map(Some).map_err(|err| { + Error::config(format!( + "invalid OPENSHELL_HOST_GATEWAY_IP value '{trimmed}': {err}" + )) + }) +} + +fn docker_gateway_route( + info: &SystemInfo, + bridge_gateway_ip: IpAddr, + port: u16, + host_gateway_ip: Option, +) -> DockerGatewayRoute { + if let Some(host_alias_ip) = host_gateway_ip { + return DockerGatewayRoute::Bridge { + bind_address: SocketAddr::new(host_alias_ip, port), + host_alias_ip, + }; + } + + if is_docker_desktop(info) { + DockerGatewayRoute::HostGateway + } else { + DockerGatewayRoute::Bridge { + bind_address: SocketAddr::new(bridge_gateway_ip, port), + host_alias_ip: bridge_gateway_ip, + } + } +} + +fn is_docker_desktop(info: &SystemInfo) -> bool { + let operating_system = info + .operating_system + .as_deref() + .unwrap_or_default() + .to_ascii_lowercase(); + if operating_system.contains("docker desktop") { + return true; + } + + info.labels.as_ref().is_some_and(|labels| { + labels + .iter() + .any(|label| label.starts_with("com.docker.desktop.")) + }) +} + +fn docker_extra_hosts(route: &DockerGatewayRoute) -> Vec { + match route { + DockerGatewayRoute::Bridge { host_alias_ip, .. } => vec![ + format!("{HOST_DOCKER_INTERNAL}:{host_alias_ip}"), + format!("{HOST_OPENSHELL_INTERNAL}:{host_alias_ip}"), + ], + DockerGatewayRoute::HostGateway => { + vec![format!("{HOST_OPENSHELL_INTERNAL}:host-gateway")] + } + } +} + +async fn ensure_bridge_network(docker: &Docker, network_name: &str) -> CoreResult { + match docker.inspect_network(network_name, None).await { + Ok(network) => return validate_bridge_network(network_name, &network), + Err(err) if !is_not_found_error(&err) => { + return Err(Error::execution(format!( + "failed to inspect Docker network '{network_name}': {err}" + ))); + } + Err(_) => {} + } + + docker + .create_network(NetworkCreateRequest { + name: network_name.to_string(), + driver: Some(DOCKER_NETWORK_DRIVER.to_string()), + attachable: Some(true), + labels: Some(HashMap::from([( + MANAGED_BY_LABEL_KEY.to_string(), + MANAGED_BY_LABEL_VALUE.to_string(), + )])), + ..Default::default() + }) + .await + .map(|_| ()) + .or_else(|err| { + if is_conflict_error(&err) { + Ok(()) + } else { + Err(Error::execution(format!( + "failed to create Docker network '{network_name}': {err}" + ))) + } + })?; + + let network = docker + .inspect_network(network_name, None) + .await + .map_err(|err| { + Error::execution(format!( + "failed to inspect Docker network '{network_name}' after create: {err}" + )) + })?; + validate_bridge_network(network_name, &network) +} + +fn validate_bridge_network( + network_name: &str, + network: &bollard::models::NetworkInspect, +) -> CoreResult { + if network.driver.as_deref() != Some(DOCKER_NETWORK_DRIVER) { + return Err(Error::config(format!( + "Docker network '{network_name}' must use the '{DOCKER_NETWORK_DRIVER}' driver, found '{}'", + network.driver.as_deref().unwrap_or("unknown") + ))); + } + + docker_bridge_gateway_ip(network_name, network) +} + +fn docker_bridge_gateway_ip( + network_name: &str, + network: &bollard::models::NetworkInspect, +) -> CoreResult { + let Some(configs) = network.ipam.as_ref().and_then(|ipam| ipam.config.as_ref()) else { + return Err(Error::config(format!( + "Docker bridge network '{network_name}' does not expose IPAM gateway configuration" + ))); + }; + + for config in configs { + let Some(gateway) = config.gateway.as_deref() else { + continue; + }; + let ip = gateway.parse::().map_err(|err| { + Error::config(format!( + "Docker bridge network '{network_name}' has invalid gateway '{gateway}': {err}" + )) + })?; + if matches!(ip, IpAddr::V4(_)) { + return Ok(ip); + } + } + + Err(Error::config(format!( + "Docker bridge network '{network_name}' does not have an IPv4 IPAM gateway" + ))) +} + +fn docker_resource_limits( + template: &DriverSandboxTemplate, +) -> Result { + let Some(resources) = template.resources.as_ref() else { + return Ok(DockerResourceLimits::default()); + }; + + if !resources.cpu_request.trim().is_empty() { + return Err(Status::failed_precondition( + "docker compute driver does not support resources.requests.cpu", + )); + } + if !resources.memory_request.trim().is_empty() { + return Err(Status::failed_precondition( + "docker compute driver does not support resources.requests.memory", + )); + } + + Ok(DockerResourceLimits { + nano_cpus: parse_cpu_limit(&resources.cpu_limit)?, + memory_bytes: parse_memory_limit(&resources.memory_limit)?, + }) +} + +#[allow(clippy::cast_possible_truncation)] +fn parse_cpu_limit(value: &str) -> Result, Status> { + let value = value.trim(); + if value.is_empty() { + return Ok(None); + } + if let Some(millicores) = value.strip_suffix('m') { + let millicores = millicores.parse::().map_err(|_| { + Status::failed_precondition(format!( + "invalid docker cpu_limit '{value}'; expected an integer or millicore quantity", + )) + })?; + if millicores <= 0 { + return Err(Status::failed_precondition( + "docker cpu_limit must be greater than zero", + )); + } + return Ok(Some(millicores.saturating_mul(1_000_000))); + } + + let cores = value.parse::().map_err(|_| { + Status::failed_precondition(format!( + "invalid docker cpu_limit '{value}'; expected an integer or millicore quantity", + )) + })?; + if !cores.is_finite() || cores <= 0.0 { + return Err(Status::failed_precondition( + "docker cpu_limit must be greater than zero", + )); + } + + Ok(Some((cores * 1_000_000_000.0).round() as i64)) +} + +#[allow(clippy::cast_possible_truncation)] +fn parse_memory_limit(value: &str) -> Result, Status> { + let value = value.trim(); + if value.is_empty() { + return Ok(None); + } + + let number_end = value + .find(|ch: char| !(ch.is_ascii_digit() || ch == '.')) + .unwrap_or(value.len()); + let (number, suffix) = value.split_at(number_end); + let amount = number.parse::().map_err(|_| { + Status::failed_precondition(format!( + "invalid docker memory_limit '{value}'; expected a Kubernetes-style quantity", + )) + })?; + if !amount.is_finite() || amount <= 0.0 { + return Err(Status::failed_precondition( + "docker memory_limit must be greater than zero", + )); + } + + let multiplier = match suffix { + "" => 1_f64, + "Ki" => 1024_f64, + "Mi" => 1024_f64.powi(2), + "Gi" => 1024_f64.powi(3), + "Ti" => 1024_f64.powi(4), + "Pi" => 1024_f64.powi(5), + "Ei" => 1024_f64.powi(6), + "K" => 1000_f64, + "M" => 1000_f64.powi(2), + "G" => 1000_f64.powi(3), + "T" => 1000_f64.powi(4), + "P" => 1000_f64.powi(5), + "E" => 1000_f64.powi(6), + _ => { + return Err(Status::failed_precondition(format!( + "invalid docker memory_limit suffix '{suffix}'", + ))); + } + }; + + Ok(Some((amount * multiplier).round() as i64)) +} + +fn sandbox_from_container_summary( + summary: &ContainerSummary, + readiness: &dyn SupervisorReadiness, +) -> Option { + let labels = summary.labels.as_ref()?; + let id = labels.get(SANDBOX_ID_LABEL_KEY)?.clone(); + let name = labels.get(SANDBOX_NAME_LABEL_KEY)?.clone(); + let namespace = labels + .get(SANDBOX_NAMESPACE_LABEL_KEY) + .cloned() + .unwrap_or_default(); + + let supervisor_connected = readiness.is_supervisor_connected(&id); + Some(DriverSandbox { + id, + name: name.clone(), + namespace, + spec: None, + status: Some(driver_status_from_summary( + summary, + &name, + supervisor_connected, + )), + }) +} + +fn driver_status_from_summary( + summary: &ContainerSummary, + sandbox_name: &str, + supervisor_connected: bool, +) -> DriverSandboxStatus { + let state = summary.state.unwrap_or(ContainerSummaryStateEnum::EMPTY); + let (ready, reason, message, deleting) = container_ready_condition(state, supervisor_connected); + + DriverSandboxStatus { + sandbox_name: summary_container_name(summary).unwrap_or_else(|| sandbox_name.to_string()), + instance_id: summary.id.clone().unwrap_or_default(), + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: vec![DriverCondition { + r#type: "Ready".to_string(), + status: ready.to_string(), + reason: reason.to_string(), + message: message.to_string(), + last_transition_time: String::new(), + }], + deleting, + } +} + +fn container_ready_condition( + state: ContainerSummaryStateEnum, + supervisor_connected: bool, +) -> (&'static str, &'static str, &'static str, bool) { + match state { + ContainerSummaryStateEnum::RUNNING => { + if supervisor_connected { + ( + "True", + "SupervisorConnected", + "Supervisor relay is live", + false, + ) + } else { + ( + "False", + "DependenciesNotReady", + "Container is running; waiting for supervisor relay", + false, + ) + } + } + ContainerSummaryStateEnum::CREATED => ("False", "Starting", "Container created", false), + ContainerSummaryStateEnum::RESTARTING => ( + "False", + "ContainerRestarting", + "Container is restarting after a failure", + false, + ), + ContainerSummaryStateEnum::EMPTY => { + ("False", "Starting", "Container state is unknown", false) + } + ContainerSummaryStateEnum::REMOVING => { + ("False", "Deleting", "Container is being removed", true) + } + ContainerSummaryStateEnum::PAUSED => { + ("False", "ContainerPaused", "Container is paused", false) + } + ContainerSummaryStateEnum::EXITED => { + ("False", "ContainerExited", "Container exited", false) + } + ContainerSummaryStateEnum::DEAD => ("False", "ContainerDead", "Container is dead", false), + } +} + +fn summary_container_name(summary: &ContainerSummary) -> Option { + summary + .names + .as_ref() + .and_then(|names| names.first()) + .map(|name| name.trim_start_matches('/').to_string()) + .filter(|name| !name.is_empty()) +} + +fn summary_container_target(summary: &ContainerSummary) -> Option { + // Prefer the container ID: it's stable while the container exists and is + // accepted by Docker APIs just like a name. Fall back to the parsed name + // for transient summaries that do not include an ID. + summary + .id + .as_deref() + .filter(|id| !id.is_empty()) + .map(str::to_string) + .or_else(|| summary_container_name(summary)) +} + +fn container_state_needs_shutdown_stop(state: ContainerSummaryStateEnum) -> bool { + matches!( + state, + ContainerSummaryStateEnum::RUNNING + | ContainerSummaryStateEnum::RESTARTING + | ContainerSummaryStateEnum::PAUSED + ) +} + +/// States from which a managed container can be brought back to running by +/// `start_container`. Skip `Restarting` (already coming up), `Removing`, +/// `Dead` (terminal), `Paused` (needs `unpause`, not `start`), and +/// `Running` (nothing to do). +fn container_state_needs_resume(state: ContainerSummaryStateEnum) -> bool { + matches!( + state, + ContainerSummaryStateEnum::EXITED | ContainerSummaryStateEnum::CREATED + ) +} + +fn docker_stop_timeout_secs(timeout_secs: u32) -> i32 { + i32::try_from(timeout_secs).unwrap_or(i32::MAX) +} + +fn emit_snapshot_diff( + events: &broadcast::Sender, + previous: &HashMap, + current: &HashMap, +) { + for (sandbox_id, sandbox) in current { + if previous.get(sandbox_id) == Some(sandbox) { + continue; + } + let _ = events.send(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { + sandbox: Some(sandbox.clone()), + }, + )), + }); + } + + for sandbox_id in previous.keys() { + if current.contains_key(sandbox_id) { + continue; + } + let _ = events.send(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Deleted( + WatchSandboxesDeletedEvent { + sandbox_id: sandbox_id.clone(), + }, + )), + }); + } +} + +fn label_filters(values: impl IntoIterator) -> HashMap> { + HashMap::from([("label".to_string(), values.into_iter().collect())]) +} + +fn managed_container_label_filters( + sandbox_namespace: &str, + extra_values: impl IntoIterator, +) -> HashMap> { + let mut values = vec![ + format!("{MANAGED_BY_LABEL_KEY}={MANAGED_BY_LABEL_VALUE}"), + format!("{SANDBOX_NAMESPACE_LABEL_KEY}={sandbox_namespace}"), + ]; + values.extend(extra_values); + label_filters(values) +} + +/// Maximum Docker container name length. Docker's own limit is 253 bytes, but +/// we cap at a conservative 200 to leave headroom for tooling that truncates +/// names further. +const MAX_CONTAINER_NAME_LEN: usize = 200; +const CONTAINER_NAME_PREFIX: &str = "openshell-"; + +fn container_name_for_sandbox(sandbox: &DriverSandbox) -> String { + let id_suffix = sanitize_docker_name(&sandbox.id); + let name = sanitize_docker_name(&sandbox.name); + if name.is_empty() { + let mut base = format!("{CONTAINER_NAME_PREFIX}{id_suffix}"); + // The prefix is always < MAX_CONTAINER_NAME_LEN. Truncate the id + // suffix only if the sandbox id itself is pathologically long. + if base.len() > MAX_CONTAINER_NAME_LEN { + base.truncate(MAX_CONTAINER_NAME_LEN); + } + return base; + } + + // Reserve space for the prefix and the `-` tail so the id + // suffix — which is what makes the name unique between sandboxes that + // share a human-readable prefix — is never truncated away. + let reserved = CONTAINER_NAME_PREFIX.len() + 1 + id_suffix.len(); + if reserved >= MAX_CONTAINER_NAME_LEN { + // Pathological sandbox id. Fall back to `` and truncate. + let mut base = format!("{CONTAINER_NAME_PREFIX}{id_suffix}"); + base.truncate(MAX_CONTAINER_NAME_LEN); + return trim_container_name_tail(base); + } + + let name_budget = MAX_CONTAINER_NAME_LEN - reserved; + let truncated_name = if name.len() > name_budget { + trim_container_name_tail(name[..name_budget].to_string()) + } else { + name + }; + format!("{CONTAINER_NAME_PREFIX}{truncated_name}-{id_suffix}") +} + +/// Docker container names may not end with `-`, `.`, or `_`. Truncation can +/// leave one of those trailing, so strip them before returning. +fn trim_container_name_tail(mut value: String) -> String { + while value + .chars() + .last() + .is_some_and(|ch| matches!(ch, '-' | '.' | '_')) + { + value.pop(); + } + value +} + +fn sanitize_docker_name(value: &str) -> String { + value + .chars() + .map(|ch| { + if ch.is_ascii_alphanumeric() || matches!(ch, '_' | '.' | '-') { + ch.to_ascii_lowercase() + } else { + '-' + } + }) + .collect::() + .trim_matches('-') + .to_string() +} + +fn normalize_docker_arch(arch: &str) -> String { + match arch { + "x86_64" => "amd64".to_string(), + "aarch64" => "arm64".to_string(), + other => other.to_ascii_lowercase(), + } +} + +pub(crate) async fn resolve_supervisor_bin( + docker: &Docker, + docker_config: &DockerComputeConfig, + daemon_arch: &str, +) -> CoreResult { + // Tier 1: explicit --docker-supervisor-bin / OPENSHELL_DOCKER_SUPERVISOR_BIN. + if let Some(path) = docker_config.supervisor_bin.clone() { + let path = canonicalize_existing_file(&path, "docker supervisor binary")?; + validate_linux_elf_binary(&path)?; + return Ok(path); + } + + // Tier 2: sibling `openshell-sandbox` next to the running gateway + // (release artifact layout). Linux-only because the sibling must be a + // Linux ELF to bind-mount into a Linux container. + if cfg!(target_os = "linux") { + let current_exe = std::env::current_exe() + .map_err(|err| Error::config(format!("failed to resolve current executable: {err}")))?; + if let Some(parent) = current_exe.parent() { + let sibling = parent.join("openshell-sandbox"); + if sibling.is_file() { + let path = canonicalize_existing_file(&sibling, "docker supervisor binary")?; + if validate_linux_elf_binary(&path).is_ok() { + return Ok(path); + } + } + } + } + + // Tier 3: local cargo target build (developer workflow). Preferred + // over a registry pull when available because it matches whatever the + // developer just built. + let target_candidates = linux_supervisor_candidates(daemon_arch); + for candidate in &target_candidates { + if candidate.is_file() { + let path = canonicalize_existing_file(candidate, "docker supervisor binary")?; + if validate_linux_elf_binary(&path).is_ok() { + return Ok(path); + } + } + } + + // Tier 4: pull the supervisor image from a registry and extract the + // binary to a host-side cache keyed by image content digest. This is + // the default path for released gateway binaries. + let image = docker_config + .supervisor_image + .clone() + .unwrap_or_else(default_docker_supervisor_image); + extract_supervisor_bin_from_image(docker, &image).await +} + +fn linux_supervisor_candidates(daemon_arch: &str) -> Vec { + match daemon_arch { + "arm64" => vec![PathBuf::from( + "target/aarch64-unknown-linux-gnu/release/openshell-sandbox", + )], + "amd64" => vec![PathBuf::from( + "target/x86_64-unknown-linux-gnu/release/openshell-sandbox", + )], + _ => Vec::new(), + } +} + +/// Pull the supervisor image (if not already local), extract +/// `/openshell-sandbox` to a host cache keyed by the image's content +/// digest, and return the cache path. +/// +/// The extraction is atomic: the binary is written to a sibling temp file +/// inside the digest-keyed directory and renamed into place, so concurrent +/// gateway starts don't observe a partial file. +async fn extract_supervisor_bin_from_image(docker: &Docker, image: &str) -> CoreResult { + // Inspect first to see if the image is already present; only pull on miss. + let inspect = match docker.inspect_image(image).await { + Ok(inspect) => inspect, + Err(err) if is_not_found_error(&err) => { + info!(image = image, "Pulling docker supervisor image"); + pull_supervisor_image(docker, image).await?; + docker.inspect_image(image).await.map_err(|err| { + Error::config(format!( + "failed to inspect docker supervisor image '{image}' after pull: {err}", + )) + })? + } + Err(err) => { + return Err(Error::config(format!( + "failed to inspect docker supervisor image '{image}': {err}", + ))); + } + }; + + let digest = inspect.id.clone().ok_or_else(|| { + Error::config(format!( + "docker supervisor image '{image}' inspect response has no Id", + )) + })?; + + let cache_path = supervisor_cache_path(&digest)?; + if cache_path.is_file() { + validate_linux_elf_binary(&cache_path)?; + return Ok(cache_path); + } + + let cache_dir = cache_path.parent().ok_or_else(|| { + Error::config(format!( + "docker supervisor cache path '{}' has no parent directory", + cache_path.display(), + )) + })?; + std::fs::create_dir_all(cache_dir).map_err(|err| { + Error::config(format!( + "failed to create docker supervisor cache dir '{}': {err}", + cache_dir.display(), + )) + })?; + + info!( + image = image, + digest = digest, + cache_path = %cache_path.display(), + "Extracting supervisor binary from image to host cache", + ); + + let binary_bytes = extract_supervisor_binary_bytes(docker, image).await?; + write_cache_binary_atomic(&cache_path, &binary_bytes)?; + validate_linux_elf_binary(&cache_path)?; + Ok(cache_path) +} + +async fn pull_supervisor_image(docker: &Docker, image: &str) -> CoreResult<()> { + let mut stream = docker.create_image( + Some(CreateImageOptions { + from_image: Some(image.to_string()), + ..Default::default() + }), + None, + None, + ); + while let Some(result) = stream.next().await { + result.map_err(|err| { + Error::config(format!( + "failed to pull docker supervisor image '{image}': {err}", + )) + })?; + } + Ok(()) +} + +/// Create a short-lived container from `image`, stream out the supervisor +/// binary as a tar archive, and return the untarred file bytes. The +/// container is always removed, even on error paths. +async fn extract_supervisor_binary_bytes(docker: &Docker, image: &str) -> CoreResult> { + let container_name = temp_extract_container_name(); + docker + .create_container( + Some( + CreateContainerOptionsBuilder::default() + .name(container_name.as_str()) + .build(), + ), + ContainerCreateBody { + image: Some(image.to_string()), + entrypoint: Some(vec!["/openshell-sandbox".to_string()]), + cmd: Some(Vec::new()), + ..Default::default() + }, + ) + .await + .map_err(|err| { + Error::config(format!( + "failed to create extractor container from '{image}': {err}", + )) + })?; + + // Always tear down the extractor container, even if extraction fails. + let result = download_binary_from_container(docker, &container_name).await; + if let Err(remove_err) = docker + .remove_container( + &container_name, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await + { + warn!( + container = container_name, + error = %remove_err, + "Failed to remove supervisor extractor container", + ); + } + result +} + +async fn download_binary_from_container( + docker: &Docker, + container_name: &str, +) -> CoreResult> { + let options = DownloadFromContainerOptionsBuilder::default() + .path(SUPERVISOR_IMAGE_BINARY_PATH) + .build(); + let mut stream = docker.download_from_container(container_name, Some(options)); + + let mut tar_bytes = Vec::new(); + while let Some(chunk) = stream.next().await { + let chunk: Bytes = chunk.map_err(|err| { + Error::config(format!( + "failed to read supervisor binary stream from '{container_name}': {err}", + )) + })?; + tar_bytes.extend_from_slice(&chunk); + } + + extract_first_tar_entry(&tar_bytes).map_err(|err| { + Error::config(format!( + "failed to extract supervisor binary from tar archive returned by '{container_name}': {err}", + )) + }) +} + +/// Extract the payload of the first regular-file entry in a tar archive. +/// Docker's `/containers//archive` endpoint returns a single-file tar +/// when `path` points to a file, so we only need the first entry. +fn extract_first_tar_entry(tar_bytes: &[u8]) -> Result, String> { + let mut archive = tar::Archive::new(std::io::Cursor::new(tar_bytes)); + let mut entries = archive + .entries() + .map_err(|err| format!("open tar archive: {err}"))?; + let mut entry = entries + .next() + .ok_or_else(|| "tar archive was empty".to_string())? + .map_err(|err| format!("read tar entry: {err}"))?; + let mut bytes = Vec::new(); + entry + .read_to_end(&mut bytes) + .map_err(|err| format!("read tar entry payload: {err}"))?; + Ok(bytes) +} + +fn write_cache_binary_atomic(final_path: &Path, bytes: &[u8]) -> CoreResult<()> { + let dir = final_path.parent().ok_or_else(|| { + Error::config(format!( + "docker supervisor cache path '{}' has no parent directory", + final_path.display(), + )) + })?; + let mut temp = tempfile::Builder::new() + .prefix(".openshell-sandbox-") + .tempfile_in(dir) + .map_err(|err| { + Error::config(format!( + "failed to create temp file for supervisor binary in '{}': {err}", + dir.display(), + )) + })?; + std::io::Write::write_all(&mut temp, bytes).map_err(|err| { + Error::config(format!( + "failed to write supervisor binary to temp file: {err}", + )) + })?; + temp.as_file().sync_all().map_err(|err| { + Error::config(format!("failed to sync supervisor binary temp file: {err}")) + })?; + + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(temp.path(), std::fs::Permissions::from_mode(0o755)).map_err( + |err| { + Error::config(format!( + "failed to chmod supervisor binary temp file: {err}", + )) + }, + )?; + } + + temp.persist(final_path).map_err(|err| { + Error::config(format!( + "failed to rename supervisor binary into '{}': {}", + final_path.display(), + err.error, + )) + })?; + Ok(()) +} + +/// Cache path for an extracted supervisor binary, keyed by the image's +/// content-addressable digest (e.g. `sha256:abc123…`). The digest-prefixed +/// directory keeps stale extractions from earlier releases isolated so they +/// can be GC'd without affecting the active binary. +fn supervisor_cache_path(digest: &str) -> CoreResult { + let base = openshell_core::paths::xdg_data_dir() + .map_err(|err| Error::config(format!("failed to resolve XDG data dir: {err}")))?; + Ok(supervisor_cache_path_with_base(&base, digest)) +} + +fn supervisor_cache_path_with_base(base: &Path, digest: &str) -> PathBuf { + let sanitized = digest.replace(':', "-"); + base.join("openshell") + .join("docker-supervisor") + .join(sanitized) + .join("openshell-sandbox") +} + +fn temp_extract_container_name() -> String { + use std::sync::atomic::{AtomicU64, Ordering}; + static SEQ: AtomicU64 = AtomicU64::new(0); + let pid = std::process::id(); + let seq = SEQ.fetch_add(1, Ordering::Relaxed); + format!("openshell-supervisor-extract-{pid}-{seq}") +} + +fn canonicalize_existing_file(path: &Path, description: &str) -> CoreResult { + if !path.is_file() { + return Err(Error::config(format!( + "{description} '{}' does not exist or is not a file", + path.display() + ))); + } + std::fs::canonicalize(path).map_err(|err| { + Error::config(format!( + "failed to resolve {description} '{}': {err}", + path.display() + )) + }) +} + +pub(crate) fn validate_linux_elf_binary(path: &Path) -> CoreResult<()> { + let mut file = std::fs::File::open(path).map_err(|err| { + Error::config(format!( + "failed to open docker supervisor binary '{}': {err}", + path.display() + )) + })?; + let mut magic = [0_u8; 4]; + file.read_exact(&mut magic).map_err(|err| { + Error::config(format!( + "failed to read docker supervisor binary '{}': {err}", + path.display() + )) + })?; + if magic != [0x7f, b'E', b'L', b'F'] { + return Err(Error::config(format!( + "docker supervisor binary '{}' must be a Linux ELF executable", + path.display() + ))); + } + Ok(()) +} + +pub(crate) fn docker_guest_tls_paths( + config: &Config, + docker_config: &DockerComputeConfig, +) -> CoreResult> { + let tls_flags_provided = docker_config.guest_tls_ca.is_some() + || docker_config.guest_tls_cert.is_some() + || docker_config.guest_tls_key.is_some(); + + if !config.grpc_endpoint.starts_with("https://") { + if tls_flags_provided { + return Err(Error::config(format!( + "--docker-tls-ca/--docker-tls-cert/--docker-tls-key were provided but OPENSHELL_GRPC_ENDPOINT is '{}'; TLS materials require an https:// endpoint", + config.grpc_endpoint, + ))); + } + return Ok(None); + } + + let provided = [ + docker_config.guest_tls_ca.as_ref(), + docker_config.guest_tls_cert.as_ref(), + docker_config.guest_tls_key.as_ref(), + ]; + if provided.iter().all(Option::is_none) { + return Err(Error::config( + "docker compute driver requires --docker-tls-ca, --docker-tls-cert, and --docker-tls-key when OPENSHELL_GRPC_ENDPOINT uses https://", + )); + } + + let Some(ca) = docker_config.guest_tls_ca.clone() else { + return Err(Error::config( + "--docker-tls-ca is required when Docker sandbox TLS materials are configured", + )); + }; + let Some(cert) = docker_config.guest_tls_cert.clone() else { + return Err(Error::config( + "--docker-tls-cert is required when Docker sandbox TLS materials are configured", + )); + }; + let Some(key) = docker_config.guest_tls_key.clone() else { + return Err(Error::config( + "--docker-tls-key is required when Docker sandbox TLS materials are configured", + )); + }; + + Ok(Some(DockerGuestTlsPaths { + ca: canonicalize_existing_file(&ca, "docker TLS CA certificate")?, + cert: canonicalize_existing_file(&cert, "docker TLS client certificate")?, + key: canonicalize_existing_file(&key, "docker TLS client private key")?, + })) +} + +fn is_not_found_error(err: &BollardError) -> bool { + matches!( + err, + BollardError::DockerResponseServerError { + status_code: 404, + .. + } + ) +} + +fn is_conflict_error(err: &BollardError) -> bool { + matches!( + err, + BollardError::DockerResponseServerError { + status_code: 409, + .. + } + ) +} + +fn is_not_modified_error(err: &BollardError) -> bool { + matches!( + err, + BollardError::DockerResponseServerError { + status_code: 304, + .. + } + ) +} + +fn create_status_from_docker_error(operation: &str, err: BollardError) -> Status { + if matches!( + err, + BollardError::DockerResponseServerError { + status_code: 409, + .. + } + ) { + Status::already_exists("sandbox already exists") + } else { + internal_status(operation, err) + } +} + +fn internal_status(operation: &str, err: BollardError) -> Status { + Status::internal(format!("{operation} failed: {err}")) +} + +#[cfg(test)] +mod tests; diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs new file mode 100644 index 0000000000..e41f2688e4 --- /dev/null +++ b/crates/openshell-driver-docker/src/tests.rs @@ -0,0 +1,841 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use super::*; +use openshell_core::config::DEFAULT_SERVER_PORT; +use openshell_core::proto::compute::v1::{ + DriverResourceRequirements, DriverSandboxSpec, DriverSandboxTemplate, +}; +use std::fs; +use std::net::{IpAddr, Ipv4Addr, SocketAddr}; +use tempfile::TempDir; + +const TLS_MOUNT_DIR: &str = "/etc/openshell/tls/client"; + +fn test_sandbox() -> DriverSandbox { + // Mirrors the gateway-supplied request: the public `Sandbox` API no + // longer carries `namespace`, so the gateway elides the field and the + // driver must source it from its own runtime config. + DriverSandbox { + id: "sbx-123".to_string(), + name: "demo".to_string(), + namespace: String::new(), + spec: Some(DriverSandboxSpec { + log_level: "debug".to_string(), + environment: HashMap::from([("SPEC_ENV".to_string(), "spec".to_string())]), + template: Some(DriverSandboxTemplate { + image: "ghcr.io/nvidia/openshell/sandbox:dev".to_string(), + agent_socket_path: String::new(), + labels: HashMap::new(), + environment: HashMap::from([("TEMPLATE_ENV".to_string(), "template".to_string())]), + resources: None, + platform_config: None, + }), + gpu: false, + gpu_device: String::new(), + }), + status: None, + } +} + +fn runtime_config() -> DockerDriverRuntimeConfig { + DockerDriverRuntimeConfig { + default_image: "image:latest".to_string(), + image_pull_policy: String::new(), + sandbox_namespace: "default".to_string(), + grpc_endpoint: "https://localhost:8443".to_string(), + network_name: DEFAULT_DOCKER_NETWORK_NAME.to_string(), + gateway_route: DockerGatewayRoute::Bridge { + bind_address: SocketAddr::new( + IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), + DEFAULT_SERVER_PORT, + ), + host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), + }, + ssh_socket_path: "/run/openshell/ssh.sock".to_string(), + stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS, + log_level: "info".to_string(), + supervisor_bin: PathBuf::from("/tmp/openshell-sandbox"), + guest_tls: Some(DockerGuestTlsPaths { + ca: PathBuf::from("/tmp/ca.crt"), + cert: PathBuf::from("/tmp/tls.crt"), + key: PathBuf::from("/tmp/tls.key"), + }), + daemon_version: "28.0.0".to_string(), + supports_gpu: false, + } +} + +#[test] +fn container_visible_endpoint_rewrites_loopback_hosts() { + assert_eq!( + docker_container_openshell_endpoint( + "https://localhost:8443", + HOST_OPENSHELL_INTERNAL, + DEFAULT_SERVER_PORT, + ), + "https://host.openshell.internal:8080/" + ); + assert_eq!( + docker_container_openshell_endpoint( + "http://127.0.0.1:8080", + HOST_OPENSHELL_INTERNAL, + DEFAULT_SERVER_PORT, + ), + "http://host.openshell.internal:8080/" + ); + assert_eq!( + docker_container_openshell_endpoint( + "https://gateway.internal:8443", + HOST_OPENSHELL_INTERNAL, + DEFAULT_SERVER_PORT, + ), + "https://host.openshell.internal:8080/" + ); +} + +#[test] +fn docker_bridge_gateway_ip_requires_ipv4_gateway() { + let network = bollard::models::NetworkInspect { + driver: Some(DOCKER_NETWORK_DRIVER.to_string()), + ipam: Some(bollard::models::Ipam { + config: Some(vec![ + bollard::models::IpamConfig { + gateway: Some("fd00::1".to_string()), + ..Default::default() + }, + bollard::models::IpamConfig { + gateway: Some("172.18.0.1".to_string()), + ..Default::default() + }, + ]), + ..Default::default() + }), + ..Default::default() + }; + + assert_eq!( + docker_bridge_gateway_ip(DEFAULT_DOCKER_NETWORK_NAME, &network).unwrap(), + IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)) + ); + + let ipv6_only_network = bollard::models::NetworkInspect { + driver: Some(DOCKER_NETWORK_DRIVER.to_string()), + ipam: Some(bollard::models::Ipam { + config: Some(vec![bollard::models::IpamConfig { + gateway: Some("fd00::1".to_string()), + ..Default::default() + }]), + ..Default::default() + }), + ..Default::default() + }; + + assert!( + docker_bridge_gateway_ip(DEFAULT_DOCKER_NETWORK_NAME, &ipv6_only_network) + .unwrap_err() + .to_string() + .contains("IPv4 IPAM gateway") + ); +} + +#[test] +fn docker_gateway_route_uses_host_gateway_for_docker_desktop() { + let info = SystemInfo { + operating_system: Some("Docker Desktop".to_string()), + labels: Some(vec![ + "com.docker.desktop.address=unix:///tmp/docker.sock".to_string(), + ]), + ..Default::default() + }; + + assert_eq!( + docker_gateway_route( + &info, + IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), + DEFAULT_SERVER_PORT, + None, + ), + DockerGatewayRoute::HostGateway + ); + assert_eq!( + docker_extra_hosts(&DockerGatewayRoute::HostGateway), + vec!["host.openshell.internal:host-gateway".to_string()] + ); +} + +#[test] +fn docker_gateway_route_uses_bridge_gateway_for_linux_docker() { + let info = SystemInfo { + operating_system: Some("Ubuntu 24.04 LTS".to_string()), + ..Default::default() + }; + + let route = docker_gateway_route( + &info, + IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), + DEFAULT_SERVER_PORT, + None, + ); + + assert_eq!( + route, + DockerGatewayRoute::Bridge { + bind_address: "172.18.0.1:8080".parse().unwrap(), + host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), + } + ); + assert_eq!( + docker_extra_hosts(&route), + vec![ + "host.docker.internal:172.18.0.1".to_string(), + "host.openshell.internal:172.18.0.1".to_string() + ] + ); +} + +#[test] +fn docker_gateway_route_prefers_configured_host_gateway_ip() { + let info = SystemInfo { + operating_system: Some("Ubuntu 24.04 LTS".to_string()), + ..Default::default() + }; + + let route = docker_gateway_route( + &info, + IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), + DEFAULT_SERVER_PORT, + Some(IpAddr::V4(Ipv4Addr::new(172, 20, 0, 4))), + ); + + assert_eq!( + route, + DockerGatewayRoute::Bridge { + bind_address: "172.20.0.4:8080".parse().unwrap(), + host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 20, 0, 4)), + } + ); + assert_eq!( + docker_extra_hosts(&route), + vec![ + "host.docker.internal:172.20.0.4".to_string(), + "host.openshell.internal:172.20.0.4".to_string() + ] + ); +} + +#[test] +fn parse_optional_host_gateway_ip_rejects_invalid_values() { + assert_eq!(parse_optional_host_gateway_ip("").unwrap(), None); + assert_eq!( + parse_optional_host_gateway_ip("172.20.0.4").unwrap(), + Some(IpAddr::V4(Ipv4Addr::new(172, 20, 0, 4))) + ); + assert!( + parse_optional_host_gateway_ip("not-an-ip") + .unwrap_err() + .to_string() + .contains("OPENSHELL_HOST_GATEWAY_IP") + ); +} + +#[test] +fn parse_cpu_limit_supports_cores_and_millicores() { + assert_eq!(parse_cpu_limit("250m").unwrap(), Some(250_000_000)); + assert_eq!(parse_cpu_limit("2").unwrap(), Some(2_000_000_000)); + assert!(parse_cpu_limit("0").is_err()); +} + +#[test] +fn parse_memory_limit_supports_binary_quantities() { + assert_eq!(parse_memory_limit("512Mi").unwrap(), Some(536_870_912)); + assert_eq!(parse_memory_limit("1G").unwrap(), Some(1_000_000_000)); + assert!(parse_memory_limit("12XB").is_err()); +} + +#[test] +fn docker_resource_limits_rejects_requests() { + let template = DriverSandboxTemplate { + image: "img".to_string(), + agent_socket_path: String::new(), + labels: HashMap::new(), + environment: HashMap::new(), + resources: Some(DriverResourceRequirements { + cpu_request: "250m".to_string(), + cpu_limit: String::new(), + memory_request: String::new(), + memory_limit: String::new(), + }), + platform_config: None, + }; + + let err = docker_resource_limits(&template).unwrap_err(); + assert_eq!(err.code(), tonic::Code::FailedPrecondition); + assert!(err.message().contains("resources.requests.cpu")); +} + +#[test] +fn build_environment_sets_docker_tls_paths() { + let env = build_environment(&test_sandbox(), &runtime_config()); + assert!(env.contains(&format!("OPENSHELL_TLS_CA={TLS_CA_MOUNT_PATH}"))); + assert!(env.contains(&format!("OPENSHELL_TLS_CERT={TLS_CERT_MOUNT_PATH}"))); + assert!(env.contains(&format!("OPENSHELL_TLS_KEY={TLS_KEY_MOUNT_PATH}"))); + assert!(env.contains(&"TEMPLATE_ENV=template".to_string())); + assert!(env.contains(&"SPEC_ENV=spec".to_string())); + assert!(env.contains(&"OPENSHELL_SANDBOX_COMMAND=sleep infinity".to_string())); + assert!( + !env.iter() + .any(|entry| entry.starts_with("OPENSHELL_SSH_HANDSHAKE_SECRET=")) + ); + assert!( + !env.iter() + .any(|entry| entry.starts_with("OPENSHELL_SSH_HANDSHAKE_SKEW_SECS=")) + ); +} + +#[test] +fn build_environment_keeps_path_driver_controlled() { + let mut sandbox = test_sandbox(); + let spec = sandbox.spec.as_mut().unwrap(); + spec.environment + .insert("PATH".to_string(), "/malicious/spec/bin".to_string()); + spec.template + .as_mut() + .unwrap() + .environment + .insert("PATH".to_string(), "/malicious/template/bin".to_string()); + + let env = build_environment(&sandbox, &runtime_config()); + let path_entries = env + .iter() + .filter(|entry| entry.starts_with("PATH=")) + .collect::>(); + + let expected_path = format!("PATH={SUPERVISOR_PATH}"); + assert_eq!(path_entries.len(), 1); + assert_eq!(path_entries[0], &expected_path); +} + +#[test] +fn build_mounts_uses_docker_tls_directory() { + let mounts = build_mounts(&runtime_config()); + let targets = mounts + .iter() + .filter_map(|mount| mount.target.clone()) + .collect::>(); + assert!(targets.contains(&SUPERVISOR_MOUNT_PATH.to_string())); + assert!(targets.contains(&TLS_CA_MOUNT_PATH.to_string())); + assert!(targets.contains(&TLS_CERT_MOUNT_PATH.to_string())); + assert!(targets.contains(&TLS_KEY_MOUNT_PATH.to_string())); + assert!( + targets + .iter() + .all(|target| target.starts_with(TLS_MOUNT_DIR) || target == SUPERVISOR_MOUNT_PATH) + ); +} + +#[test] +fn managed_container_label_filters_include_gateway_namespace() { + let filters = + managed_container_label_filters("tenant-a", [format!("{SANDBOX_ID_LABEL_KEY}=sbx-123")]); + let labels = filters.get("label").unwrap(); + + assert!(labels.contains(&format!("{MANAGED_BY_LABEL_KEY}={MANAGED_BY_LABEL_VALUE}"))); + assert!(labels.contains(&format!("{SANDBOX_NAMESPACE_LABEL_KEY}=tenant-a"))); + assert!(labels.contains(&format!("{SANDBOX_ID_LABEL_KEY}=sbx-123"))); +} + +#[test] +fn build_container_create_body_clears_inherited_cmd() { + let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); + + assert_eq!( + create_body.entrypoint, + Some(vec![SUPERVISOR_MOUNT_PATH.to_string()]) + ); + assert_eq!(create_body.cmd, Some(Vec::new())); + assert_eq!( + create_body + .labels + .as_ref() + .and_then(|labels| labels.get(SANDBOX_NAMESPACE_LABEL_KEY)), + Some(&"default".to_string()) + ); + let host_config = create_body.host_config.as_ref().unwrap(); + assert!( + host_config.device_requests.as_ref().is_none(), + "non-GPU containers should not request Docker devices" + ); + assert_eq!( + host_config.security_opt.as_ref(), + Some(&vec!["apparmor=unconfined".to_string()]) + ); + assert_eq!( + host_config.network_mode.as_deref(), + Some(DEFAULT_DOCKER_NETWORK_NAME) + ); + assert_eq!( + host_config.extra_hosts.as_ref(), + Some(&vec![ + "host.docker.internal:172.18.0.1".to_string(), + "host.openshell.internal:172.18.0.1".to_string() + ]) + ); + assert_eq!( + create_body + .networking_config + .as_ref() + .and_then(|config| config.endpoints_config.as_ref()) + .and_then(|endpoints| endpoints.get(DEFAULT_DOCKER_NETWORK_NAME)), + Some(&EndpointSettings::default()) + ); +} + +#[test] +fn validate_sandbox_rejects_gpu_when_cdi_unavailable() { + let config = runtime_config(); + let mut sandbox = test_sandbox(); + sandbox.spec.as_mut().unwrap().gpu = true; + + let err = DockerComputeDriver::validate_sandbox(&sandbox, &config).unwrap_err(); + + assert_eq!(err.code(), tonic::Code::FailedPrecondition); + assert!(err.message().contains("Docker CDI")); +} + +#[test] +fn build_container_create_body_maps_gpu_to_all_cdi_device() { + let mut config = runtime_config(); + config.supports_gpu = true; + let mut sandbox = test_sandbox(); + sandbox.spec.as_mut().unwrap().gpu = true; + + let create_body = build_container_create_body(&sandbox, &config).unwrap(); + let request = create_body + .host_config + .as_ref() + .and_then(|host_config| host_config.device_requests.as_ref()) + .and_then(|requests| requests.first()) + .expect("GPU request should add a Docker device request"); + + assert_eq!(request.driver.as_deref(), Some("cdi")); + assert_eq!( + request.device_ids.as_ref().unwrap(), + &vec![CDI_GPU_DEVICE_ALL.to_string()] + ); +} + +#[test] +fn require_sandbox_identifier_rejects_when_id_and_name_are_empty() { + // Regression test: `delete_sandbox` (and the other identifier-keyed + // RPCs) must refuse requests where both the id and the name are + // empty. Otherwise the empty filters fed to + // `find_managed_container_summary` match the first managed container + // in the namespace, allowing an arbitrary sandbox to be deleted. + let err = require_sandbox_identifier("", "").unwrap_err(); + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains("sandbox_id or sandbox_name")); + + require_sandbox_identifier("sbx-1", "").expect("id-only is accepted"); + require_sandbox_identifier("", "demo").expect("name-only is accepted"); + require_sandbox_identifier("sbx-1", "demo").expect("id and name is accepted"); +} + +#[test] +fn build_container_create_body_uses_bridge_network() { + let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); + let host_config = create_body.host_config.expect("host_config is populated"); + + assert_eq!( + host_config.network_mode, + Some(DEFAULT_DOCKER_NETWORK_NAME.to_string()), + "sandbox should join the driver-managed bridge network" + ); + assert_eq!( + host_config.extra_hosts, + Some(vec![ + "host.docker.internal:172.18.0.1".to_string(), + "host.openshell.internal:172.18.0.1".to_string() + ]), + "sandbox should expose stable host aliases for gateway callbacks" + ); +} + +#[test] +fn build_container_create_body_uses_runtime_namespace_label() { + // Regression test: the namespace label must come from the driver's + // runtime config, not from `DriverSandbox.namespace`. The gateway + // does not populate `DriverSandbox.namespace`, so a container created + // with that empty value would not match subsequent list/get/find + // queries (which filter on `config.sandbox_namespace`), leaking + // sandboxes that the driver itself cannot observe. + let mut config = runtime_config(); + config.sandbox_namespace = "tenant-a".to_string(); + let mut sandbox = test_sandbox(); + sandbox.namespace = "ignored-by-driver".to_string(); + + let create_body = build_container_create_body(&sandbox, &config).unwrap(); + let labels = create_body.labels.expect("labels are populated"); + + assert_eq!( + labels.get(SANDBOX_NAMESPACE_LABEL_KEY), + Some(&"tenant-a".to_string()), + "namespace label must reflect the driver's runtime config" + ); +} + +#[test] +fn driver_status_keeps_running_sandboxes_provisioning_with_stable_message() { + let running = ContainerSummary { + id: Some("cid".to_string()), + names: Some(vec!["/openshell-demo".to_string()]), + labels: Some(HashMap::from([ + (SANDBOX_ID_LABEL_KEY.to_string(), "sbx-1".to_string()), + (SANDBOX_NAME_LABEL_KEY.to_string(), "demo".to_string()), + ( + SANDBOX_NAMESPACE_LABEL_KEY.to_string(), + "default".to_string(), + ), + ])), + state: Some(ContainerSummaryStateEnum::RUNNING), + status: Some("Up 2 seconds".to_string()), + ..Default::default() + }; + let exited = ContainerSummary { + state: Some(ContainerSummaryStateEnum::EXITED), + status: Some("Exited (1) 3 seconds ago".to_string()), + ..running.clone() + }; + let running_later = ContainerSummary { + status: Some("Up 4 seconds".to_string()), + ..running.clone() + }; + + let running_status = driver_status_from_summary(&running, "demo", false); + let running_later_status = driver_status_from_summary(&running_later, "demo", false); + assert_eq!(running_status.conditions[0].status, "False"); + assert_eq!(running_status.conditions[0].reason, "DependenciesNotReady"); + assert_eq!( + running_status.conditions[0].message, + "Container is running; waiting for supervisor relay" + ); + assert_eq!(running_status.conditions, running_later_status.conditions); + + let exited_status = driver_status_from_summary(&exited, "demo", false); + assert_eq!(exited_status.conditions[0].status, "False"); + assert_eq!(exited_status.conditions[0].reason, "ContainerExited"); + assert_eq!(exited_status.conditions[0].message, "Container exited"); + + // With a live supervisor session, a RUNNING container flips Ready=True + // so ExecSandbox and other "sandbox must be ready" gates can proceed. + let running_connected = driver_status_from_summary(&running, "demo", true); + assert_eq!(running_connected.conditions[0].status, "True"); + assert_eq!( + running_connected.conditions[0].reason, + "SupervisorConnected" + ); + + // Supervisor readiness is ignored for non-RUNNING states -- an exited + // container must not report Ready=True. + let exited_connected = driver_status_from_summary(&exited, "demo", true); + assert_eq!(exited_connected.conditions[0].status, "False"); +} + +#[test] +fn driver_status_marks_restarting_sandboxes_as_error() { + let restarting = ContainerSummary { + id: Some("cid".to_string()), + names: Some(vec!["/openshell-demo".to_string()]), + labels: Some(HashMap::from([ + (SANDBOX_ID_LABEL_KEY.to_string(), "sbx-1".to_string()), + (SANDBOX_NAME_LABEL_KEY.to_string(), "demo".to_string()), + ( + SANDBOX_NAMESPACE_LABEL_KEY.to_string(), + "default".to_string(), + ), + ])), + state: Some(ContainerSummaryStateEnum::RESTARTING), + status: Some("Restarting (1) 2 seconds ago".to_string()), + ..Default::default() + }; + + let status = driver_status_from_summary(&restarting, "demo", false); + assert_eq!(status.conditions[0].status, "False"); + assert_eq!(status.conditions[0].reason, "ContainerRestarting"); + assert_eq!( + status.conditions[0].message, + "Container is restarting after a failure" + ); +} + +#[test] +fn validate_linux_elf_binary_rejects_non_elf_files() { + let tempdir = TempDir::new().unwrap(); + let path = tempdir.path().join("openshell-sandbox"); + fs::write(&path, b"not-elf").unwrap(); + + let err = validate_linux_elf_binary(&path).unwrap_err(); + assert!(err.to_string().contains("Linux ELF executable")); +} + +#[test] +fn docker_guest_tls_paths_require_all_files_for_https() { + let config = Config::new(None).with_grpc_endpoint("https://localhost:8443"); + let tempdir = TempDir::new().unwrap(); + let ca = tempdir.path().join("ca.crt"); + fs::write(&ca, b"ca").unwrap(); + + let err = docker_guest_tls_paths( + &config, + &DockerComputeConfig { + guest_tls_ca: Some(ca), + ..Default::default() + }, + ) + .unwrap_err(); + assert!(err.to_string().contains("--docker-tls-cert")); +} + +#[test] +fn linux_supervisor_candidates_follow_daemon_arch() { + assert_eq!( + linux_supervisor_candidates("amd64"), + vec![PathBuf::from( + "target/x86_64-unknown-linux-gnu/release/openshell-sandbox", + )] + ); + assert_eq!( + linux_supervisor_candidates("arm64"), + vec![PathBuf::from( + "target/aarch64-unknown-linux-gnu/release/openshell-sandbox", + )] + ); +} + +#[test] +fn container_name_preserves_id_suffix_for_long_names() { + // Names up to 253 chars are permitted by the gRPC layer. The id + // suffix is what makes the container name unique between sandboxes + // sharing a prefix, so it must always appear in the final name. + let long_name = "a".repeat(253); + let first = DriverSandbox { + id: "sbx-first-1234567890".to_string(), + name: long_name, + namespace: "default".to_string(), + spec: None, + status: None, + }; + let second = DriverSandbox { + id: "sbx-second-0987654321".to_string(), + ..first.clone() + }; + + let first_container = container_name_for_sandbox(&first); + let second_container = container_name_for_sandbox(&second); + + assert!( + first_container.len() <= MAX_CONTAINER_NAME_LEN, + "container name {} exceeded {MAX_CONTAINER_NAME_LEN} chars: {first_container}", + first_container.len(), + ); + assert!( + first_container.ends_with(&first.id), + "container name should end with sandbox id: {first_container}", + ); + assert_ne!( + first_container, second_container, + "container names must differ for sandboxes with distinct ids", + ); +} + +#[test] +fn container_name_empty_sandbox_name_uses_id_only() { + let sandbox = DriverSandbox { + id: "sbx-abc".to_string(), + name: String::new(), + namespace: "default".to_string(), + spec: None, + status: None, + }; + assert_eq!(container_name_for_sandbox(&sandbox), "openshell-sbx-abc",); +} + +#[test] +fn trim_container_name_tail_strips_separators() { + assert_eq!(trim_container_name_tail("foo-".to_string()), "foo"); + assert_eq!(trim_container_name_tail("foo-.".to_string()), "foo"); + assert_eq!(trim_container_name_tail("foo_-.".to_string()), "foo"); + assert_eq!(trim_container_name_tail("foo".to_string()), "foo"); +} + +#[test] +fn docker_guest_tls_paths_rejects_tls_flags_without_https() { + let config = Config::new(None).with_grpc_endpoint("http://localhost:8080"); + let tempdir = TempDir::new().unwrap(); + let ca = tempdir.path().join("ca.crt"); + fs::write(&ca, b"ca").unwrap(); + + let err = docker_guest_tls_paths( + &config, + &DockerComputeConfig { + guest_tls_ca: Some(ca), + ..Default::default() + }, + ) + .unwrap_err(); + assert!(err.to_string().contains("https://")); +} + +#[test] +fn docker_guest_tls_paths_allows_plain_http_without_tls_flags() { + let config = Config::new(None).with_grpc_endpoint("http://localhost:8080"); + let result = docker_guest_tls_paths(&config, &DockerComputeConfig::default()).unwrap(); + assert!(result.is_none()); +} + +#[test] +fn default_docker_supervisor_image_uses_nvidia_ghcr_repo() { + let image = default_docker_supervisor_image(); + assert!( + image.starts_with("ghcr.io/nvidia/openshell/supervisor:"), + "unexpected default image reference: {image}", + ); +} + +#[test] +fn docker_supervisor_image_tag_prefers_explicit_build_tags() { + assert_eq!( + resolve_default_docker_supervisor_image_tag(Some("1.2.3"), Some("sha"), "0.0.0"), + "1.2.3", + ); + assert_eq!( + resolve_default_docker_supervisor_image_tag(None, Some("sha"), "0.0.0"), + "sha", + ); + assert_eq!( + resolve_default_docker_supervisor_image_tag(None, None, "1.2.3"), + "1.2.3", + ); + assert_eq!( + resolve_default_docker_supervisor_image_tag(Some(""), Some(""), "0.0.0"), + "dev", + ); +} + +#[test] +fn supervisor_cache_path_namespaces_by_digest_under_openshell_data_dir() { + let base = PathBuf::from("/var/cache/share"); + let path = + supervisor_cache_path_with_base(&base, "sha256:abc123deadbeef0123456789cafe0123456789fe"); + + assert_eq!( + path, + PathBuf::from( + "/var/cache/share/openshell/docker-supervisor/sha256-abc123deadbeef0123456789cafe0123456789fe/openshell-sandbox", + ), + ); +} + +#[test] +fn supervisor_cache_path_isolates_different_digests() { + let base = PathBuf::from("/data"); + let left = supervisor_cache_path_with_base(&base, "sha256:aaaaaaaa"); + let right = supervisor_cache_path_with_base(&base, "sha256:bbbbbbbb"); + assert_ne!( + left.parent().unwrap(), + right.parent().unwrap(), + "digest-keyed directories must differ so rollouts are isolated", + ); +} + +#[test] +fn write_cache_binary_atomic_materializes_file_with_executable_mode() { + let tempdir = TempDir::new().unwrap(); + let target = tempdir.path().join("nested").join("openshell-sandbox"); + fs::create_dir_all(target.parent().unwrap()).unwrap(); + + write_cache_binary_atomic(&target, b"\x7fELFpayload").unwrap(); + + assert!(target.is_file()); + assert_eq!(fs::read(&target).unwrap(), b"\x7fELFpayload"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let mode = fs::metadata(&target).unwrap().permissions().mode() & 0o777; + assert_eq!(mode, 0o755, "expected 0755, got {mode:04o}"); + } +} + +#[test] +fn write_cache_binary_atomic_overwrites_existing_file() { + let tempdir = TempDir::new().unwrap(); + let target = tempdir.path().join("openshell-sandbox"); + fs::write(&target, b"stale").unwrap(); + + write_cache_binary_atomic(&target, b"\x7fELFfresh").unwrap(); + assert_eq!(fs::read(&target).unwrap(), b"\x7fELFfresh"); +} + +#[test] +fn temp_extract_container_names_are_unique_per_call() { + let first = temp_extract_container_name(); + let second = temp_extract_container_name(); + assert_ne!(first, second); + assert!(first.starts_with("openshell-supervisor-extract-")); +} + +#[test] +fn extract_first_tar_entry_returns_payload_of_single_file_archive() { + // Build a tar archive with the same shape Docker returns from + // `/containers//archive` for a single file. + let payload = b"\x7fELFtest-binary-bytes"; + let mut tar_buf = Vec::new(); + { + let mut builder = tar::Builder::new(&mut tar_buf); + let mut header = tar::Header::new_gnu(); + header.set_path("openshell-sandbox").unwrap(); + header.set_size(payload.len() as u64); + header.set_mode(0o755); + header.set_cksum(); + builder.append(&header, payload.as_slice()).unwrap(); + builder.finish().unwrap(); + } + + let extracted = extract_first_tar_entry(&tar_buf).unwrap(); + assert_eq!(extracted, payload); +} + +#[test] +fn extract_first_tar_entry_rejects_empty_archive() { + let mut tar_buf = Vec::new(); + tar::Builder::new(&mut tar_buf).finish().unwrap(); + let err = extract_first_tar_entry(&tar_buf).unwrap_err(); + assert!(err.contains("empty"), "unexpected error message: {err}"); +} + +#[test] +fn container_state_needs_resume_matches_startable_states() { + for state in [ + ContainerSummaryStateEnum::EXITED, + ContainerSummaryStateEnum::CREATED, + ] { + assert!( + container_state_needs_resume(state), + "{state:?} should be resumed with Docker start", + ); + } + + for state in [ + ContainerSummaryStateEnum::RUNNING, + ContainerSummaryStateEnum::RESTARTING, + ContainerSummaryStateEnum::PAUSED, + ContainerSummaryStateEnum::DEAD, + ContainerSummaryStateEnum::REMOVING, + ContainerSummaryStateEnum::EMPTY, + ] { + assert!( + !container_state_needs_resume(state), + "{state:?} should not be resumed with Docker start", + ); + } +} diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index 3ce98eae89..838262c771 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -6,9 +6,15 @@ pub struct KubernetesComputeConfig { pub namespace: String, pub default_image: String, pub image_pull_policy: String, + /// Image that provides the `openshell-sandbox` supervisor binary. + /// An init container copies the binary from this image into a shared + /// emptyDir volume before the sandbox container starts. + pub supervisor_image: String, + /// Kubernetes `imagePullPolicy` for the supervisor init container. + /// Empty string delegates to the Kubernetes default. + pub supervisor_image_pull_policy: String, pub grpc_endpoint: String, - pub ssh_listen_addr: String, - pub ssh_port: u16, + pub ssh_socket_path: String, pub ssh_handshake_secret: String, pub ssh_handshake_skew_secs: u64, pub client_tls_secret_name: String, diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index 440703af57..b0c2e59178 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -5,7 +5,7 @@ use crate::config::KubernetesComputeConfig; use futures::{Stream, StreamExt, TryStreamExt}; -use k8s_openapi::api::core::v1::{Event as KubeEventObj, Node, Pod}; +use k8s_openapi::api::core::v1::{Event as KubeEventObj, Node}; use kube::api::{Api, ApiResource, DeleteParams, ListParams, PostParams}; use kube::core::gvk::GroupVersionKind; use kube::core::{DynamicObject, ObjectMeta}; @@ -15,12 +15,10 @@ use openshell_core::proto::compute::v1::{ DriverCondition as SandboxCondition, DriverPlatformEvent as PlatformEvent, DriverSandbox as Sandbox, DriverSandboxSpec as SandboxSpec, DriverSandboxStatus as SandboxStatus, DriverSandboxTemplate as SandboxTemplate, - GetCapabilitiesResponse, ResolveSandboxEndpointResponse, SandboxEndpoint, - WatchSandboxesDeletedEvent, WatchSandboxesEvent, WatchSandboxesPlatformEvent, - WatchSandboxesSandboxEvent, sandbox_endpoint, watch_sandboxes_event, + GetCapabilitiesResponse, WatchSandboxesDeletedEvent, WatchSandboxesEvent, + WatchSandboxesPlatformEvent, WatchSandboxesSandboxEvent, watch_sandboxes_event, }; use std::collections::BTreeMap; -use std::net::IpAddr; use std::pin::Pin; use std::time::Duration; use tokio::sync::mpsc; @@ -104,6 +102,7 @@ const WORKSPACE_SENTINEL: &str = ".workspace-initialized"; #[derive(Clone)] pub struct KubernetesComputeDriver { client: Client, + watch_client: Client, config: KubernetesComputeConfig, } @@ -119,17 +118,30 @@ impl std::fmt::Debug for KubernetesComputeDriver { impl KubernetesComputeDriver { pub async fn new(config: KubernetesComputeConfig) -> Result { - let mut kube_config = match kube::Config::incluster() { + let base_config = match kube::Config::incluster() { Ok(c) => c, Err(_) => kube::Config::infer() .await .map_err(kube::Error::InferConfig)?, }; + + let mut kube_config = base_config.clone(); kube_config.connect_timeout = Some(Duration::from_secs(10)); kube_config.read_timeout = Some(Duration::from_secs(30)); kube_config.write_timeout = Some(Duration::from_secs(30)); let client = Client::try_from(kube_config)?; - Ok(Self { client, config }) + + let mut watch_kube_config = base_config; + watch_kube_config.connect_timeout = Some(Duration::from_secs(10)); + watch_kube_config.read_timeout = None; + watch_kube_config.write_timeout = Some(Duration::from_secs(30)); + let watch_client = Client::try_from(watch_kube_config)?; + + Ok(Self { + client, + watch_client, + config, + }) } pub async fn capabilities(&self) -> Result { @@ -138,6 +150,7 @@ impl KubernetesComputeDriver { driver_version: openshell_core::VERSION.to_string(), default_image: self.config.default_image.clone(), supports_gpu: self.has_gpu_capacity().await.unwrap_or(false), + gpu_count: 0, }) } @@ -149,14 +162,20 @@ impl KubernetesComputeDriver { &self.config.namespace } - pub fn ssh_listen_addr(&self) -> &str { - &self.config.ssh_listen_addr + pub fn ssh_socket_path(&self) -> &str { + &self.config.ssh_socket_path } pub const fn ssh_handshake_skew_secs(&self) -> u64 { self.config.ssh_handshake_skew_secs } + fn watch_api(&self) -> Api { + let gvk = GroupVersionKind::gvk(SANDBOX_GROUP, SANDBOX_VERSION, SANDBOX_KIND); + let resource = ApiResource::from_gvk(&gvk); + Api::namespaced_with(self.watch_client.clone(), &self.config.namespace, &resource) + } + fn api(&self) -> Api { let gvk = GroupVersionKind::gvk(SANDBOX_GROUP, SANDBOX_VERSION, SANDBOX_KIND); let resource = ApiResource::from_gvk(&gvk); @@ -271,21 +290,6 @@ impl KubernetesComputeDriver { &self.config.ssh_handshake_secret } - async fn agent_pod_ip(&self, pod_name: &str) -> Result, KubeError> { - let api: Api = Api::namespaced(self.client.clone(), &self.config.namespace); - match api.get(pod_name).await { - Ok(pod) => { - let ip = pod - .status - .and_then(|status| status.pod_ip) - .and_then(|ip| ip.parse().ok()); - Ok(ip) - } - Err(KubeError::Api(err)) if err.code == 404 => Ok(None), - Err(err) => Err(err), - } - } - pub async fn create_sandbox(&self, sandbox: &Sandbox) -> Result<(), KubernetesDriverError> { let name = sandbox.name.as_str(); info!( @@ -308,10 +312,12 @@ impl KubernetesComputeDriver { sandbox.spec.as_ref(), &self.config.default_image, &self.config.image_pull_policy, + &self.config.supervisor_image, + &self.config.supervisor_image_pull_policy, &sandbox.id, &sandbox.name, &self.config.grpc_endpoint, - self.ssh_listen_addr(), + self.ssh_socket_path(), self.ssh_handshake_secret(), self.ssh_handshake_skew_secs(), &self.config.client_tls_secret_name, @@ -407,56 +413,12 @@ impl KubernetesComputeDriver { } } - pub async fn resolve_sandbox_endpoint( - &self, - sandbox: &Sandbox, - ) -> Result { - if let Some(status) = sandbox.status.as_ref() - && !status.instance_id.is_empty() - { - match self.agent_pod_ip(&status.instance_id).await { - Ok(Some(ip)) => { - return Ok(ResolveSandboxEndpointResponse { - endpoint: Some(SandboxEndpoint { - target: Some(sandbox_endpoint::Target::Ip(ip.to_string())), - port: u32::from(self.config.ssh_port), - }), - }); - } - Ok(None) => { - return Err(KubernetesDriverError::Precondition( - "sandbox agent pod IP is not available".to_string(), - )); - } - Err(err) => { - return Err(KubernetesDriverError::Message(format!( - "failed to resolve agent pod IP: {err}" - ))); - } - } - } - - if sandbox.name.is_empty() { - return Err(KubernetesDriverError::Precondition( - "sandbox has no name".to_string(), - )); - } - - Ok(ResolveSandboxEndpointResponse { - endpoint: Some(SandboxEndpoint { - target: Some(sandbox_endpoint::Target::Host(format!( - "{}.{}.svc.cluster.local", - sandbox.name, self.config.namespace - ))), - port: u32::from(self.config.ssh_port), - }), - }) - } - + // Kept `async` to match the gRPC handler signature in `grpc.rs`, which awaits this method. + #[allow(clippy::unused_async)] pub async fn watch_sandboxes(&self) -> Result { let namespace = self.config.namespace.clone(); - let sandbox_api = self.api(); - let event_api: Api = Api::namespaced(self.client.clone(), &namespace); + let sandbox_api = self.watch_api(); + let event_api: Api = Api::namespaced(self.watch_client.clone(), &namespace); let mut sandbox_stream = watcher::watcher(sandbox_api, watcher::Config::default()).boxed(); let mut event_stream = watcher::watcher(event_api, watcher::Config::default()).boxed(); let (tx, rx) = mpsc::channel(256); @@ -622,7 +584,6 @@ fn sandbox_from_object(namespace: &str, obj: DynamicObject) -> Result serde_json::Value { serde_json::json!({ "name": SUPERVISOR_VOLUME_NAME, - "hostPath": { - "path": SUPERVISOR_HOST_PATH, - "type": "DirectoryOrCreate" - } + "emptyDir": {} }) } @@ -731,26 +686,61 @@ fn supervisor_volume_mount() -> serde_json::Value { }) } -/// Apply supervisor side-load transforms to an already-built pod template JSON. +/// Build the init container that copies the supervisor binary into the emptyDir. /// -/// This injects the hostPath volume, volume mount, command override, and -/// `runAsUser: 0` into the pod template, targeting the `agent` container -/// (or the first container if no `agent` is found). +/// The supervisor image is expected to have `openshell-sandbox` on its PATH +/// (e.g. at `/usr/local/bin/openshell-sandbox`). The init container resolves +/// the binary via `command -v` and copies it into the shared emptyDir volume +/// so the agent container can execute it from a fixed, writable path. +fn supervisor_init_container( + supervisor_image: &str, + supervisor_image_pull_policy: &str, +) -> serde_json::Value { + let copy_cmd = format!( + "set -e && \ + mkdir -p {SUPERVISOR_MOUNT_PATH} && \ + SUPERVISOR=$(command -v openshell-sandbox) && \ + cp \"$SUPERVISOR\" {SUPERVISOR_MOUNT_PATH}/openshell-sandbox && \ + chmod +x {SUPERVISOR_MOUNT_PATH}/openshell-sandbox" + ); + let mut spec = serde_json::json!({ + "name": SUPERVISOR_INIT_CONTAINER_NAME, + "image": supervisor_image, + "command": ["sh", "-c", copy_cmd], + "securityContext": {"runAsUser": 0}, + "volumeMounts": [{ + "name": SUPERVISOR_VOLUME_NAME, + "mountPath": SUPERVISOR_MOUNT_PATH, + "readOnly": false + }] + }); + if !supervisor_image_pull_policy.is_empty() { + spec["imagePullPolicy"] = serde_json::json!(supervisor_image_pull_policy); + } + spec +} + +/// Apply supervisor side-load transforms to an already-built pod template JSON. /// -/// The supervisor binary is always side-loaded from the k3s node filesystem -/// via a read-only hostPath volume. No init container is needed. +/// Injects an emptyDir volume, an init container that copies the supervisor +/// binary from the supervisor image into that volume, and a read-only volume +/// mount + command override on the agent container. /// /// The `runAsUser: 0` override ensures the supervisor binary runs as root /// regardless of the image's `USER` directive. The supervisor needs root for /// network namespace creation, proxy setup, and Landlock/seccomp configuration. /// It drops to the appropriate non-root user for child processes via the /// policy's `run_as_user`/`run_as_group`. -fn apply_supervisor_sideload(pod_template: &mut serde_json::Value) { +fn apply_supervisor_sideload( + pod_template: &mut serde_json::Value, + supervisor_image: &str, + supervisor_image_pull_policy: &str, +) { let Some(spec) = pod_template.get_mut("spec").and_then(|v| v.as_object_mut()) else { return; }; - // 1. Add the hostPath volume to spec.volumes + // 1. Add the emptyDir volume to spec.volumes let volumes = spec .entry("volumes") .or_insert_with(|| serde_json::json!([])) @@ -759,7 +749,19 @@ fn apply_supervisor_sideload(pod_template: &mut serde_json::Value) { volumes.push(supervisor_volume()); } - // 2. Find the agent container and add volume mount + command override + // 2. Add the init container that copies the binary into the emptyDir + let init_containers = spec + .entry("initContainers") + .or_insert_with(|| serde_json::json!([])) + .as_array_mut(); + if let Some(init_containers) = init_containers { + init_containers.push(supervisor_init_container( + supervisor_image, + supervisor_image_pull_policy, + )); + } + + // 3. Find the agent container and add volume mount + command override let Some(containers) = spec.get_mut("containers").and_then(|v| v.as_array_mut()) else { return; }; @@ -812,7 +814,7 @@ fn apply_supervisor_sideload(pod_template: &mut serde_json::Value) { /// /// The PVC volume itself is **not** added here — the Sandbox CRD controller /// automatically creates a volume for each entry in `volumeClaimTemplates` -/// (following the StatefulSet convention). Adding one here would create a +/// (following the `StatefulSet` convention). Adding one here would create a /// duplicate volume name and fail pod validation. /// /// The init container mounts the PVC at a temporary path so it can still see @@ -922,10 +924,12 @@ fn sandbox_to_k8s_spec( spec: Option<&SandboxSpec>, default_image: &str, image_pull_policy: &str, + supervisor_image: &str, + supervisor_image_pull_policy: &str, sandbox_id: &str, sandbox_name: &str, grpc_endpoint: &str, - ssh_listen_addr: &str, + ssh_socket_path: &str, ssh_handshake_secret: &str, ssh_handshake_skew_secs: u64, client_tls_secret_name: &str, @@ -962,10 +966,12 @@ fn sandbox_to_k8s_spec( spec.gpu, default_image, image_pull_policy, + supervisor_image, + supervisor_image_pull_policy, sandbox_id, sandbox_name, grpc_endpoint, - ssh_listen_addr, + ssh_socket_path, ssh_handshake_secret, ssh_handshake_skew_secs, &spec.environment, @@ -1008,10 +1014,12 @@ fn sandbox_to_k8s_spec( spec.as_ref().is_some_and(|s| s.gpu), default_image, image_pull_policy, + supervisor_image, + supervisor_image_pull_policy, sandbox_id, sandbox_name, grpc_endpoint, - ssh_listen_addr, + ssh_socket_path, ssh_handshake_secret, ssh_handshake_skew_secs, spec_env, @@ -1033,10 +1041,12 @@ fn sandbox_template_to_k8s( gpu: bool, default_image: &str, image_pull_policy: &str, + supervisor_image: &str, + supervisor_image_pull_policy: &str, sandbox_id: &str, sandbox_name: &str, grpc_endpoint: &str, - ssh_listen_addr: &str, + ssh_socket_path: &str, ssh_handshake_secret: &str, ssh_handshake_skew_secs: u64, spec_environment: &std::collections::HashMap, @@ -1044,9 +1054,6 @@ fn sandbox_template_to_k8s( host_gateway_ip: &str, inject_workspace: bool, ) -> serde_json::Value { - // The supervisor binary is always side-loaded from the node filesystem - // via a hostPath volume, regardless of which sandbox image is used. - let mut metadata = serde_json::Map::new(); if !template.labels.is_empty() { metadata.insert("labels".to_string(), serde_json::json!(template.labels)); @@ -1089,7 +1096,7 @@ fn sandbox_template_to_k8s( sandbox_id, sandbox_name, grpc_endpoint, - ssh_listen_addr, + ssh_socket_path, ssh_handshake_secret, ssh_handshake_skew_secs, !client_tls_secret_name.is_empty(), @@ -1163,8 +1170,9 @@ fn sandbox_template_to_k8s( let mut result = serde_json::Value::Object(template_value); - // Always side-load the supervisor binary from the node filesystem - apply_supervisor_sideload(&mut result); + // Side-load the supervisor binary via an init container that copies it + // from the supervisor image into a shared emptyDir volume. + apply_supervisor_sideload(&mut result, supervisor_image, supervisor_image_pull_policy); // Inject workspace persistence (init container + PVC volume mount) so // that /sandbox data survives pod rescheduling. Skipped when the user @@ -1201,10 +1209,7 @@ fn container_resources(template: &SandboxTemplate, gpu: bool) -> Option, sandbox_id: &str, sandbox_name: &str, grpc_endpoint: &str, - ssh_listen_addr: &str, + ssh_socket_path: &str, ssh_handshake_secret: &str, ssh_handshake_skew_secs: u64, tls_enabled: bool, @@ -1283,8 +1291,8 @@ fn apply_required_env( upsert_env(env, "OPENSHELL_SANDBOX", sandbox_name); upsert_env(env, "OPENSHELL_ENDPOINT", grpc_endpoint); upsert_env(env, "OPENSHELL_SANDBOX_COMMAND", "sleep infinity"); - if !ssh_listen_addr.is_empty() { - upsert_env(env, "OPENSHELL_SSH_LISTEN_ADDR", ssh_listen_addr); + if !ssh_socket_path.is_empty() { + upsert_env(env, "OPENSHELL_SSH_SOCKET_PATH", ssh_socket_path); } upsert_env(env, "OPENSHELL_SSH_HANDSHAKE_SECRET", ssh_handshake_secret); upsert_env( @@ -1478,7 +1486,7 @@ mod tests { } }); - apply_supervisor_sideload(&mut pod_template); + apply_supervisor_sideload(&mut pod_template, "custom-image:latest", "IfNotPresent"); let sc = &pod_template["spec"]["containers"][0]["securityContext"]; assert_eq!(sc["runAsUser"], 0, "runAsUser must be 0 for supervisor"); @@ -1502,7 +1510,7 @@ mod tests { } }); - apply_supervisor_sideload(&mut pod_template); + apply_supervisor_sideload(&mut pod_template, "supervisor-image:latest", "IfNotPresent"); let sc = &pod_template["spec"]["containers"][0]["securityContext"]; assert_eq!( @@ -1512,7 +1520,7 @@ mod tests { } #[test] - fn supervisor_sideload_injects_hostpath_volume_and_mount() { + fn supervisor_sideload_injects_emptydir_volume_init_container_and_mount() { let mut pod_template = serde_json::json!({ "spec": { "containers": [{ @@ -1522,33 +1530,38 @@ mod tests { } }); - apply_supervisor_sideload(&mut pod_template); - - // No init containers should be present (hostPath, not emptyDir+init) - assert!( - pod_template["spec"]["initContainers"].is_null(), - "hostPath sideload should not create init containers" - ); + apply_supervisor_sideload(&mut pod_template, "supervisor-image:latest", "IfNotPresent"); - // Volume should be a hostPath volume + // Volume should be an emptyDir let volumes = pod_template["spec"]["volumes"] .as_array() .expect("volumes should exist"); assert_eq!(volumes.len(), 1); assert_eq!(volumes[0]["name"], SUPERVISOR_VOLUME_NAME); - assert_eq!(volumes[0]["hostPath"]["path"], SUPERVISOR_HOST_PATH); - assert_eq!(volumes[0]["hostPath"]["type"], "DirectoryOrCreate"); + assert!( + volumes[0]["emptyDir"].is_object(), + "volume should be emptyDir, not hostPath" + ); - // Agent container command should be overridden + // Init container should use the supervisor image, not the sandbox image + let init_containers = pod_template["spec"]["initContainers"] + .as_array() + .expect("initContainers should exist"); + assert_eq!(init_containers.len(), 1); + assert_eq!(init_containers[0]["name"], SUPERVISOR_INIT_CONTAINER_NAME); + assert_eq!(init_containers[0]["image"], "supervisor-image:latest"); + assert_eq!(init_containers[0]["imagePullPolicy"], "IfNotPresent"); + + // Agent container command should be overridden to the emptyDir path let command = pod_template["spec"]["containers"][0]["command"] .as_array() .expect("command should be set"); assert_eq!( command[0].as_str().unwrap(), - format!("{}/openshell-sandbox", SUPERVISOR_MOUNT_PATH) + format!("{SUPERVISOR_MOUNT_PATH}/openshell-sandbox") ); - // Volume mount should be read-only + // Agent volume mount should be read-only let mounts = pod_template["spec"]["containers"][0]["volumeMounts"] .as_array() .expect("volumeMounts should exist"); @@ -1613,6 +1626,8 @@ mod tests { true, "openshell/sandbox:latest", "", + "openshell/supervisor:latest", + "", "sandbox-id", "sandbox-name", "https://gateway.example.com", @@ -1639,13 +1654,12 @@ mod tests { fn gpu_sandbox_uses_template_runtime_class_name_when_set() { let template = SandboxTemplate { platform_config: Some(Struct { - fields: [( + fields: std::iter::once(( "runtime_class_name".to_string(), Value { kind: Some(Kind::StringValue("kata-containers".to_string())), }, - )] - .into_iter() + )) .collect(), }), ..SandboxTemplate::default() @@ -1656,6 +1670,8 @@ mod tests { true, "openshell/sandbox:latest", "", + "openshell/supervisor:latest", + "", "sandbox-id", "sandbox-name", "https://gateway.example.com", @@ -1678,13 +1694,12 @@ mod tests { fn non_gpu_sandbox_uses_template_runtime_class_name_when_set() { let template = SandboxTemplate { platform_config: Some(Struct { - fields: [( + fields: std::iter::once(( "runtime_class_name".to_string(), Value { kind: Some(Kind::StringValue("kata-containers".to_string())), }, - )] - .into_iter() + )) .collect(), }), ..SandboxTemplate::default() @@ -1695,6 +1710,8 @@ mod tests { false, "openshell/sandbox:latest", "", + "openshell/supervisor:latest", + "", "sandbox-id", "sandbox-name", "https://gateway.example.com", @@ -1729,6 +1746,8 @@ mod tests { true, "openshell/sandbox:latest", "", + "openshell/supervisor:latest", + "", "sandbox-id", "sandbox-name", "https://gateway.example.com", @@ -1756,6 +1775,8 @@ mod tests { false, "openshell/sandbox:latest", "", + "openshell/supervisor:latest", + "", "sandbox-id", "sandbox-name", "https://gateway.example.com", @@ -1787,6 +1808,8 @@ mod tests { false, "openshell/sandbox:latest", "", + "openshell/supervisor:latest", + "", "sandbox-id", "sandbox-name", "https://gateway.example.com", @@ -1813,6 +1836,8 @@ mod tests { false, "openshell/sandbox:latest", "", + "openshell/supervisor:latest", + "", "sandbox-id", "sandbox-name", "https://gateway.example.com", @@ -1893,9 +1918,7 @@ mod tests { // Verify we did not inject one (which would cause a duplicate). let has_pvc_vol = pod_template["spec"]["volumes"] .as_array() - .map_or(false, |vols| { - vols.iter().any(|v| v["name"] == WORKSPACE_VOLUME_NAME) - }); + .is_some_and(|vols| vols.iter().any(|v| v["name"] == WORKSPACE_VOLUME_NAME)); assert!( !has_pvc_vol, "apply_workspace_persistence must NOT add a PVC volume (the CRD controller does that)" @@ -1958,6 +1981,8 @@ mod tests { false, "openshell/sandbox:latest", "", + "openshell/supervisor:latest", + "", "sandbox-id", "sandbox-name", "https://gateway.example.com", @@ -1970,21 +1995,21 @@ mod tests { false, // user provided custom VCTs ); - // No init container should be present + // Only the supervisor init container should be present — no workspace init container + let init_containers = pod_template["spec"]["initContainers"] + .as_array() + .expect("supervisor init container should always be present"); assert!( - pod_template["spec"]["initContainers"].is_null() - || pod_template["spec"]["initContainers"] - .as_array() - .is_none_or(|a| a.is_empty()), + !init_containers + .iter() + .any(|c| c["name"] == WORKSPACE_INIT_CONTAINER_NAME), "workspace init container must NOT be present when inject_workspace is false" ); // No workspace volume mount on agent let has_workspace_mount = pod_template["spec"]["containers"][0]["volumeMounts"] .as_array() - .map_or(false, |mounts| { - mounts.iter().any(|m| m["name"] == WORKSPACE_VOLUME_NAME) - }); + .is_some_and(|mounts| mounts.iter().any(|m| m["name"] == WORKSPACE_VOLUME_NAME)); assert!( !has_workspace_mount, "workspace mount must NOT be present when inject_workspace is false" diff --git a/crates/openshell-driver-kubernetes/src/grpc.rs b/crates/openshell-driver-kubernetes/src/grpc.rs index 2c5a94467b..51488f694d 100644 --- a/crates/openshell-driver-kubernetes/src/grpc.rs +++ b/crates/openshell-driver-kubernetes/src/grpc.rs @@ -1,12 +1,13 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +#![allow(clippy::result_large_err)] // gRPC handlers return Result<_, tonic::Status> + use futures::{Stream, StreamExt}; use openshell_core::proto::compute::v1::{ CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, GetCapabilitiesRequest, GetCapabilitiesResponse, GetSandboxRequest, GetSandboxResponse, - ListSandboxesRequest, ListSandboxesResponse, ResolveSandboxEndpointRequest, - ResolveSandboxEndpointResponse, StopSandboxRequest, StopSandboxResponse, + ListSandboxesRequest, ListSandboxesResponse, StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesEvent, WatchSandboxesRequest, compute_driver_server::ComputeDriver, }; @@ -128,21 +129,6 @@ impl ComputeDriver for ComputeDriverService { Ok(Response::new(DeleteSandboxResponse { deleted })) } - async fn resolve_sandbox_endpoint( - &self, - request: Request, - ) -> Result, Status> { - let sandbox = request - .into_inner() - .sandbox - .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; - self.driver - .resolve_sandbox_endpoint(&sandbox) - .await - .map(Response::new) - .map_err(status_from_driver_error) - } - type WatchSandboxesStream = Pin> + Send + 'static>>; diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index 76c567f593..26d323f563 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -39,8 +39,12 @@ struct Args { #[arg(long, env = "OPENSHELL_GRPC_ENDPOINT")] grpc_endpoint: Option, - #[arg(long, env = "OPENSHELL_SANDBOX_SSH_PORT", default_value_t = 2222)] - sandbox_ssh_port: u16, + #[arg( + long, + env = "OPENSHELL_SANDBOX_SSH_SOCKET_PATH", + default_value = "/run/openshell/ssh.sock" + )] + sandbox_ssh_socket_path: String, #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SECRET")] ssh_handshake_secret: String, @@ -53,6 +57,12 @@ struct Args { #[arg(long, env = "OPENSHELL_HOST_GATEWAY_IP")] host_gateway_ip: Option, + + #[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE")] + supervisor_image: Option, + + #[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY")] + supervisor_image_pull_policy: Option, } #[tokio::main] @@ -68,9 +78,12 @@ async fn main() -> Result<()> { namespace: args.sandbox_namespace, default_image: args.sandbox_image.unwrap_or_default(), image_pull_policy: args.sandbox_image_pull_policy.unwrap_or_default(), + supervisor_image: args + .supervisor_image + .unwrap_or_else(|| openshell_core::config::DEFAULT_SUPERVISOR_IMAGE.to_string()), + supervisor_image_pull_policy: args.supervisor_image_pull_policy.unwrap_or_default(), grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), - ssh_listen_addr: format!("0.0.0.0:{}", args.sandbox_ssh_port), - ssh_port: args.sandbox_ssh_port, + ssh_socket_path: args.sandbox_ssh_socket_path, ssh_handshake_secret: args.ssh_handshake_secret, ssh_handshake_skew_secs: args.ssh_handshake_skew_secs, client_tls_secret_name: args.client_tls_secret_name.unwrap_or_default(), diff --git a/crates/openshell-driver-podman/Cargo.toml b/crates/openshell-driver-podman/Cargo.toml new file mode 100644 index 0000000000..51ac698de0 --- /dev/null +++ b/crates/openshell-driver-podman/Cargo.toml @@ -0,0 +1,40 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-driver-podman" +description = "Podman compute driver for OpenShell" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[[bin]] +name = "openshell-driver-podman" +path = "src/main.rs" + +[dependencies] +openshell-core = { path = "../openshell-core" } + +tokio = { workspace = true } +tonic = { workspace = true, features = ["transport"] } +futures = { workspace = true } +tokio-stream = { workspace = true } +hyper = { workspace = true } +hyper-util = { workspace = true } +http-body-util = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +clap = { workspace = true } +nix = { workspace = true } +tracing = { workspace = true } +tracing-subscriber = { workspace = true } +thiserror = { workspace = true } +miette = { workspace = true } + +[dev-dependencies] +temp-env = "0.3" + +[lints] +workspace = true diff --git a/crates/openshell-driver-podman/src/client.rs b/crates/openshell-driver-podman/src/client.rs new file mode 100644 index 0000000000..69bfd69c04 --- /dev/null +++ b/crates/openshell-driver-podman/src/client.rs @@ -0,0 +1,855 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Thin async HTTP client for the Podman REST API over a Unix socket. + +use http_body_util::{BodyExt, Full}; +use hyper::Request; +use hyper::body::Bytes; +use hyper_util::rt::TokioIo; +use serde::de::DeserializeOwned; +use serde_json::Value; +use std::collections::HashMap; +use std::path::PathBuf; +use std::pin::Pin; +use std::time::Duration; +use tokio::net::UnixStream; +use tokio::sync::mpsc; +use tracing::debug; + +/// Podman libpod API version prefix. +const API_VERSION: &str = "v5.0.0"; + +/// Timeout for individual Podman API calls. +const API_TIMEOUT: Duration = Duration::from_secs(30); + +/// Maximum allowed size for the event stream line buffer (1 MB). +const MAX_EVENT_BUFFER: usize = 1_048_576; + +#[derive(Debug, thiserror::Error)] +pub enum PodmanApiError { + #[error("podman API not found (404): {0}")] + NotFound(String), + #[error("podman API conflict (409): {0}")] + Conflict(String), + #[error("podman API error ({status}): {message}")] + Api { status: u16, message: String }, + #[error("connection error: {0}")] + Connection(String), + #[error("timeout after {0:?}")] + Timeout(Duration), + #[error("JSON error: {0}")] + Json(String), + #[error("invalid input: {0}")] + InvalidInput(String), +} + +/// Maximum resource name length. Podman container names become directory +/// names in the storage driver, so we cap at 255 to stay within ext4/xfs +/// filename limits. +const MAX_NAME_LEN: usize = 255; + +/// Validate that a resource name is safe for URL path interpolation. +/// +/// Valid names start with an alphanumeric character and contain only +/// alphanumerics, dots, underscores, and hyphens — matching Podman's +/// own naming rules. Names longer than [`MAX_NAME_LEN`] are rejected. +pub fn validate_name(name: &str) -> Result<(), PodmanApiError> { + // Regex-equivalent: ^[a-zA-Z0-9][a-zA-Z0-9._-]*$ + if name.is_empty() { + return Err(PodmanApiError::InvalidInput( + "name must not be empty".to_string(), + )); + } + if name.len() > MAX_NAME_LEN { + return Err(PodmanApiError::InvalidInput(format!( + "name exceeds maximum length of {MAX_NAME_LEN} characters (got {})", + name.len() + ))); + } + let bytes = name.as_bytes(); + if !bytes[0].is_ascii_alphanumeric() { + return Err(PodmanApiError::InvalidInput(format!( + "name must start with an alphanumeric character: {name:?}" + ))); + } + if !bytes + .iter() + .all(|&b| b.is_ascii_alphanumeric() || b == b'.' || b == b'_' || b == b'-') + { + return Err(PodmanApiError::InvalidInput(format!( + "name contains invalid characters: {name:?}" + ))); + } + Ok(()) +} + +/// A container state snapshot returned by inspect APIs. +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct ContainerInspect { + pub id: String, + pub name: String, + pub state: ContainerState, + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub network_settings: NetworkSettings, + #[serde(default)] + pub config: ContainerConfig, +} + +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct ContainerState { + pub status: String, + #[allow(dead_code)] // kept for podman API compat + pub running: bool, + #[serde(default)] + pub exit_code: i64, + #[serde(rename = "OOMKilled")] + #[serde(default)] + pub oom_killed: bool, + #[serde(default)] + pub health: Option, + #[serde(default)] + pub started_at: Option, + #[serde(default)] + pub finished_at: Option, +} + +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct HealthState { + pub status: String, +} + +#[derive(Debug, Clone, Default, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct NetworkSettings { + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub networks: HashMap, + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub ports: HashMap>>, +} + +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct NetworkInfo { + #[serde(rename = "IPAddress")] + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub ip_address: String, +} + +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct PortBinding { + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub host_port: String, + #[serde(rename = "HostIp")] + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub host_ip: String, +} + +#[derive(Debug, Clone, Default, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct ContainerConfig { + #[serde(default)] + pub labels: HashMap, +} + +/// A container summary returned by the list API. +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct ContainerListEntry { + pub id: String, + #[serde(default)] + pub names: Vec, + pub state: String, + #[serde(default)] + pub labels: HashMap, + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub ports: Option>, + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub networks: Option>, + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub exit_code: i64, +} + +#[derive(Debug, Clone, serde::Deserialize)] +pub struct PortMappingEntry { + #[allow(dead_code)] // kept for podman API compat + pub host_port: u16, + #[allow(dead_code)] // kept for podman API compat + pub container_port: u16, + #[allow(dead_code)] // kept for podman API compat + pub protocol: String, + #[serde(default)] + #[allow(dead_code)] // kept for podman API compat + pub host_ip: String, +} + +/// A Podman event from the events stream. +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct PodmanEvent { + #[serde(rename = "Type")] + #[allow(dead_code)] // kept for podman API compat + pub event_type: String, + pub action: String, + #[serde(default)] + pub actor: EventActor, + #[serde(rename = "timeNano", default)] + #[allow(dead_code)] // kept for podman API compat + pub time_nano: i64, +} + +#[derive(Debug, Clone, Default, serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +pub struct EventActor { + #[serde(rename = "ID")] + #[serde(default)] + pub id: String, + #[serde(default)] + pub attributes: HashMap, +} + +/// System info response (subset of fields we care about). +#[derive(Debug, Clone, serde::Deserialize)] +pub struct SystemInfo { + pub host: HostInfo, +} + +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct HostInfo { + #[serde(default)] + pub cgroup_version: String, + #[serde(default)] + pub network_backend: String, + #[serde(default)] + pub security: SecurityInfo, +} + +/// Security-related fields from the Podman system info response. +/// +/// Podman returns `host.security.rootless: true` when the daemon is +/// running without root privileges (rootless mode). +#[derive(Debug, Clone, Default, serde::Deserialize)] +pub struct SecurityInfo { + #[serde(default)] + pub rootless: bool, +} + +// ── Client ─────────────────────────────────────────────────────────────── + +/// Async Podman REST API client communicating over a Unix socket. +#[derive(Debug, Clone)] +pub struct PodmanClient { + socket_path: PathBuf, +} + +impl PodmanClient { + /// Create a new client targeting the given socket path. + #[must_use] + pub fn new(socket_path: PathBuf) -> Self { + Self { socket_path } + } + + /// Open a new HTTP/1.1 connection to the Podman socket. + async fn connect( + &self, + ) -> Result>, PodmanApiError> { + let stream = UnixStream::connect(&self.socket_path).await.map_err(|e| { + PodmanApiError::Connection(format!("{}: {e}", self.socket_path.display())) + })?; + + let (sender, conn) = hyper::client::conn::http1::handshake(TokioIo::new(stream)) + .await + .map_err(|e| PodmanApiError::Connection(e.to_string()))?; + + tokio::spawn(async move { + if let Err(e) = conn.await { + debug!(error = %e, "Podman API connection closed"); + } + }); + + Ok(sender) + } + + // ── Request infrastructure ─────────────────────────────────────────── + + /// Build an HTTP request from components. + fn build_request( + method: hyper::Method, + path: &str, + body: Full, + content_type: Option<&str>, + ) -> Request> { + let mut builder = Request::builder() + .method(method) + .uri(format!("http://localhost{path}")) + .header("Host", "localhost"); + if let Some(ct) = content_type { + builder = builder.header("Content-Type", ct); + } + builder.body(body).expect("valid request") + } + + /// Send a pre-built HTTP request and return status + body bytes. + async fn send_request( + &self, + req: Request>, + timeout: Duration, + ) -> Result<(hyper::StatusCode, Bytes), PodmanApiError> { + let mut sender = self.connect().await?; + let response = tokio::time::timeout(timeout, sender.send_request(req)) + .await + .map_err(|_| PodmanApiError::Timeout(timeout))? + .map_err(|e| PodmanApiError::Connection(e.to_string()))?; + let status = response.status(); + let bytes = tokio::time::timeout(timeout, response.into_body().collect()) + .await + .map_err(|_| PodmanApiError::Timeout(timeout))? + .map_err(|e| PodmanApiError::Connection(e.to_string()))? + .to_bytes(); + Ok((status, bytes)) + } + + /// Perform a versioned HTTP request and return status + body bytes. + async fn request( + &self, + method: hyper::Method, + path: &str, + body: Option<&Value>, + timeout: Duration, + ) -> Result<(hyper::StatusCode, Bytes), PodmanApiError> { + let (full_body, content_type) = match body { + Some(json) => { + let payload = + serde_json::to_vec(json).map_err(|e| PodmanApiError::Json(e.to_string()))?; + (Full::new(Bytes::from(payload)), Some("application/json")) + } + None => (Full::new(Bytes::new()), None), + }; + let req = Self::build_request( + method, + &format!("/{API_VERSION}{path}"), + full_body, + content_type, + ); + self.send_request(req, timeout).await + } + + /// Perform a request and deserialize the JSON response. + async fn request_json( + &self, + method: hyper::Method, + path: &str, + body: Option<&Value>, + ) -> Result { + let (status, bytes) = self.request(method, path, body, API_TIMEOUT).await?; + if status.is_success() { + serde_json::from_slice(&bytes).map_err(|e| { + PodmanApiError::Json(format!("{e}: {}", String::from_utf8_lossy(&bytes))) + }) + } else { + Err(error_from_response(status.as_u16(), &bytes)) + } + } + + /// Perform a request that returns no meaningful body. + async fn request_ok( + &self, + method: hyper::Method, + path: &str, + body: Option<&Value>, + ) -> Result<(), PodmanApiError> { + let (status, bytes) = self.request(method, path, body, API_TIMEOUT).await?; + let code = status.as_u16(); + if status.is_success() || code == 304 { + Ok(()) + } else { + Err(error_from_response(code, &bytes)) + } + } + + /// Perform a versioned HTTP request with a raw byte body (not JSON). + async fn request_raw( + &self, + method: hyper::Method, + path: &str, + content_type: &str, + body: Bytes, + ) -> Result<(hyper::StatusCode, Bytes), PodmanApiError> { + let req = Self::build_request( + method, + &format!("/{API_VERSION}{path}"), + Full::new(body), + Some(content_type), + ); + self.send_request(req, API_TIMEOUT).await + } + + /// POST a JSON body and ignore 409 Conflict (resource already exists). + async fn create_ignore_conflict(&self, path: &str, body: &Value) -> Result<(), PodmanApiError> { + match self + .request_json::(hyper::Method::POST, path, Some(body)) + .await + { + Ok(_) | Err(PodmanApiError::Conflict(_)) => Ok(()), + Err(e) => Err(e), + } + } + + // ── Container operations ───────────────────────────────────────────── + + /// Create a container from a JSON spec. + pub async fn create_container(&self, spec: &Value) -> Result { + self.request_json(hyper::Method::POST, "/libpod/containers/create", Some(spec)) + .await + } + + /// Start a container by name or ID. + pub async fn start_container(&self, name: &str) -> Result<(), PodmanApiError> { + validate_name(name)?; + self.request_ok( + hyper::Method::POST, + &format!("/libpod/containers/{name}/start"), + None, + ) + .await + } + + /// Stop a container with a grace period in seconds. + pub async fn stop_container( + &self, + name: &str, + timeout_secs: u32, + ) -> Result<(), PodmanApiError> { + validate_name(name)?; + let http_timeout = Duration::from_secs(u64::from(timeout_secs) + 5); + let (status, bytes) = self + .request( + hyper::Method::POST, + &format!("/libpod/containers/{name}/stop?timeout={timeout_secs}"), + None, + http_timeout, + ) + .await?; + let code = status.as_u16(); + if status.is_success() || code == 304 { + Ok(()) + } else { + Err(error_from_response(code, &bytes)) + } + } + + /// Force-remove a container and its anonymous volumes. + pub async fn remove_container(&self, name: &str) -> Result<(), PodmanApiError> { + validate_name(name)?; + self.request_ok( + hyper::Method::DELETE, + &format!("/libpod/containers/{name}?force=true&v=true"), + None, + ) + .await + } + + /// Inspect a container by name or ID. + pub async fn inspect_container(&self, name: &str) -> Result { + validate_name(name)?; + self.request_json( + hyper::Method::GET, + &format!("/libpod/containers/{name}/json"), + None, + ) + .await + } + + /// List containers matching a label filter (e.g. `"openshell.managed=true"`). + pub async fn list_containers( + &self, + label_filter: &str, + ) -> Result, PodmanApiError> { + let filters = serde_json::json!({"label": [label_filter]}); + let encoded = url_encode(&filters.to_string()); + self.request_json( + hyper::Method::GET, + &format!("/libpod/containers/json?all=true&filters={encoded}"), + None, + ) + .await + } + + // ── Volume operations ──────────────────────────────────────────────── + + /// Create a named volume. Idempotent (conflict is ignored). + pub async fn create_volume(&self, name: &str) -> Result<(), PodmanApiError> { + validate_name(name)?; + self.create_ignore_conflict("/libpod/volumes/create", &serde_json::json!({"Name": name})) + .await + } + + /// Remove a named volume. Idempotent (not-found is ignored). + pub async fn remove_volume(&self, name: &str) -> Result<(), PodmanApiError> { + validate_name(name)?; + match self + .request_ok( + hyper::Method::DELETE, + &format!("/libpod/volumes/{name}"), + None, + ) + .await + { + Ok(()) | Err(PodmanApiError::NotFound(_)) => Ok(()), + Err(e) => Err(e), + } + } + + // ── Network operations ─────────────────────────────────────────────── + + /// Create a bridge network with DNS enabled. Idempotent. + pub async fn ensure_network(&self, name: &str) -> Result<(), PodmanApiError> { + validate_name(name)?; + self.create_ignore_conflict( + "/libpod/networks/create", + &serde_json::json!({ + "name": name, + "driver": "bridge", + "dns_enabled": true, + }), + ) + .await + } + + /// Inspect a network and return the gateway IP of its first subnet. + /// + /// The gateway IP is the host's address on the bridge network, used by + /// sandbox containers to call back to the gateway server. + pub async fn network_gateway_ip(&self, name: &str) -> Result, PodmanApiError> { + validate_name(name)?; + let encoded = url_encode(name); + let path = format!("/libpod/networks/{encoded}/json"); + let resp: Value = self.request_json(hyper::Method::GET, &path, None).await?; + // The response has "subnets": [{"gateway": "10.89.1.1", "subnet": "..."}] + let gateway = resp + .get("subnets") + .and_then(|s| s.as_array()) + .and_then(|arr| arr.first()) + .and_then(|sub| sub.get("gateway")) + .and_then(|g| g.as_str()) + .map(String::from); + Ok(gateway) + } + + // ── Secret operations ──────────────────────────────────────────────── + + /// Create a Podman secret with the given name and raw value. + /// + /// Idempotent: if a secret with the same name already exists it is + /// replaced (delete + recreate) so the value is always up-to-date. + pub async fn create_secret(&self, name: &str, value: &[u8]) -> Result<(), PodmanApiError> { + validate_name(name)?; + let encoded_name = url_encode(name); + let path = format!("/libpod/secrets/create?name={encoded_name}"); + let (status, bytes) = self + .request_raw( + hyper::Method::POST, + &path, + "application/octet-stream", + Bytes::copy_from_slice(value), + ) + .await?; + + match status.as_u16() { + 200 | 201 => Ok(()), + 409 => { + // Secret already exists — replace it. + self.remove_secret(name).await?; + let (status2, bytes2) = self + .request_raw( + hyper::Method::POST, + &path, + "application/octet-stream", + Bytes::copy_from_slice(value), + ) + .await?; + if status2.is_success() { + Ok(()) + } else { + Err(error_from_response(status2.as_u16(), &bytes2)) + } + } + _ => Err(error_from_response(status.as_u16(), &bytes)), + } + } + + /// Remove a Podman secret by name. Idempotent (not-found is ignored). + pub async fn remove_secret(&self, name: &str) -> Result<(), PodmanApiError> { + validate_name(name)?; + match self + .request_ok( + hyper::Method::DELETE, + &format!("/libpod/secrets/{name}"), + None, + ) + .await + { + Ok(()) | Err(PodmanApiError::NotFound(_)) => Ok(()), + Err(e) => Err(e), + } + } + + // ── Image operations ──────────────────────────────────────────────── + + /// Pull an image if it is not already present locally. + /// + /// Uses the `policy` parameter to decide whether to pull: + /// - `"always"` — always pull, even if a local copy exists + /// - `"missing"` — pull only when no local copy exists (default) + /// - `"never"` — never pull, fail if not local + /// - `"newer"` — pull only if the remote image is newer + /// + /// The pull `policy` is passed directly to Podman's API so that + /// Podman handles local-image resolution and registry fallback + /// natively. This avoids name-resolution mismatches between the + /// exists API and the local image store (e.g. `openshell/supervisor:dev` + /// vs `localhost/openshell/supervisor:dev`). + /// + /// The Podman pull endpoint streams NDJSON progress. We consume the + /// entire stream and check for an `error` field in the final object. + pub async fn pull_image(&self, reference: &str, policy: &str) -> Result<(), PodmanApiError> { + let path = format!( + "/libpod/images/pull?reference={}&policy={}", + url_encode(reference), + url_encode(policy), + ); + // Image pulls can be slow — use a generous timeout. + let pull_timeout = Duration::from_secs(600); + let (status, bytes) = self + .request(hyper::Method::POST, &path, None, pull_timeout) + .await?; + if !status.is_success() { + return Err(error_from_response(status.as_u16(), &bytes)); + } + // The response is NDJSON. Check the last line for an error field. + let body = String::from_utf8_lossy(&bytes); + if let Some(last_line) = body.lines().rfind(|l| !l.is_empty()) + && let Ok(obj) = serde_json::from_str::(last_line) + && let Some(err) = obj.get("error").and_then(|v| v.as_str()) + && !err.is_empty() + { + return Err(PodmanApiError::Api { + status: 500, + message: format!("image pull failed: {err}"), + }); + } + Ok(()) + } + + // ── System operations ──────────────────────────────────────────────── + + /// Ping the Podman API to verify connectivity. + pub async fn ping(&self) -> Result<(), PodmanApiError> { + // _ping is outside the versioned API path. + let req = Self::build_request(hyper::Method::GET, "/_ping", Full::new(Bytes::new()), None); + let (status, _) = self.send_request(req, API_TIMEOUT).await?; + if status.is_success() { + Ok(()) + } else { + Err(PodmanApiError::Api { + status: status.as_u16(), + message: "ping failed".to_string(), + }) + } + } + + /// Get system info. + pub async fn system_info(&self) -> Result { + self.request_json(hyper::Method::GET, "/libpod/info", None) + .await + } + + // ── Event streaming ────────────────────────────────────────────────── + + /// Start streaming container events filtered by label. + /// + /// Events are sent to the returned receiver. The background task runs + /// until the receiver is dropped. + pub async fn events_stream( + &self, + label_filter: &str, + ) -> Result>, PodmanApiError> { + let filters = serde_json::json!({ + "label": [label_filter], + "type": ["container"], + }); + let encoded = url_encode(&filters.to_string()); + let path = + format!("http://localhost/{API_VERSION}/libpod/events?stream=true&filters={encoded}"); + + let mut sender = self.connect().await?; + + let req = Request::builder() + .method(hyper::Method::GET) + .uri(&path) + .header("Host", "localhost") + .body(Full::new(Bytes::new())) + .map_err(|e| PodmanApiError::Connection(e.to_string()))?; + + let response = tokio::time::timeout(API_TIMEOUT, sender.send_request(req)) + .await + .map_err(|_| PodmanApiError::Timeout(API_TIMEOUT))? + .map_err(|e| PodmanApiError::Connection(e.to_string()))?; + + if !response.status().is_success() { + return Err(PodmanApiError::Api { + status: response.status().as_u16(), + message: "events stream request failed".to_string(), + }); + } + + let (tx, rx) = mpsc::channel(256); + let body = response.into_body(); + + tokio::spawn(async move { + let mut buffer = Vec::new(); + let mut body = body; + + loop { + use hyper::body::Body; + + let frame = + match std::future::poll_fn(|cx| Pin::new(&mut body).poll_frame(cx)).await { + Some(Ok(frame)) => frame, + Some(Err(e)) => { + let _ = tx + .send(Err(PodmanApiError::Connection(e.to_string()))) + .await; + break; + } + None => break, + }; + + if let Some(data) = frame.data_ref() { + buffer.extend_from_slice(data); + } + + if buffer.len() > MAX_EVENT_BUFFER { + tracing::error!("event stream buffer exceeded maximum size, disconnecting"); + let _ = tx + .send(Err(PodmanApiError::Connection( + "event buffer exceeded 1 MB limit".to_string(), + ))) + .await; + break; + } + + // Parse complete newline-delimited JSON lines. + while let Some(pos) = buffer.iter().position(|&b| b == b'\n') { + let line: Vec = buffer.drain(..=pos).collect(); + let trimmed = line.strip_suffix(b"\n").unwrap_or(&line); + if trimmed.is_empty() { + continue; + } + let event = serde_json::from_slice::(trimmed).map_err(|e| { + PodmanApiError::Json(format!("{e}: {}", String::from_utf8_lossy(trimmed))) + }); + if tx.send(event).await.is_err() { + return; + } + } + } + }); + + Ok(rx) + } +} + +// ── Helpers ────────────────────────────────────────────────────────────── + +fn error_from_response(status: u16, bytes: &Bytes) -> PodmanApiError { + let message = serde_json::from_slice::(bytes) + .ok() + .and_then(|v| { + v.get("message") + .or_else(|| v.get("cause")) + .and_then(Value::as_str) + .map(String::from) + }) + .unwrap_or_else(|| String::from_utf8_lossy(bytes).to_string()); + + match status { + 404 => PodmanApiError::NotFound(message), + 409 => PodmanApiError::Conflict(message), + _ => PodmanApiError::Api { status, message }, + } +} + +/// Minimal percent-encoding for query parameter values. +/// +/// Note: `percent-encoding` is available as a transitive dependency but is not +/// a direct dependency of this crate. Rather than adding a new dep for one +/// call site, we keep this self-contained implementation. +fn url_encode(s: &str) -> String { + s.bytes() + .map(|b| match b { + b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => { + String::from(b as char) + } + _ => format!("%{b:02X}"), + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn url_encode_encodes_special_characters() { + assert_eq!(url_encode("hello world"), "hello%20world"); + assert_eq!(url_encode("a=b&c=d"), "a%3Db%26c%3Dd"); + assert_eq!(url_encode("safe-_.~chars"), "safe-_.~chars"); + } + + #[test] + fn validate_name_accepts_valid_names() { + // alphanumeric, dots, hyphens, underscores + assert!(validate_name("my-container").is_ok()); + assert!(validate_name("my_container.v2").is_ok()); + assert!(validate_name("a").is_ok()); + assert!(validate_name("Container123").is_ok()); + } + + #[test] + fn validate_name_rejects_invalid_names() { + assert!(validate_name("").is_err()); // empty + assert!(validate_name("-leading").is_err()); // starts with dash + assert!(validate_name(".leading").is_err()); // starts with dot + assert!(validate_name("has/slash").is_err()); // path traversal + assert!(validate_name("../etc").is_err()); // path traversal + assert!(validate_name("has space").is_err()); // space + assert!(validate_name("has%20encoded").is_err()); // percent + assert!(validate_name("has?query").is_err()); // query char + } + + #[test] + fn validate_name_rejects_names_exceeding_max_length() { + let long_name = format!("a{}", "b".repeat(MAX_NAME_LEN)); + assert!(long_name.len() > MAX_NAME_LEN); + assert!(validate_name(&long_name).is_err()); + + // Exactly at the limit should be accepted. + let exact_name = "a".repeat(MAX_NAME_LEN); + assert!(validate_name(&exact_name).is_ok()); + } +} diff --git a/crates/openshell-driver-podman/src/config.rs b/crates/openshell-driver-podman/src/config.rs new file mode 100644 index 0000000000..d82b8d0b08 --- /dev/null +++ b/crates/openshell-driver-podman/src/config.rs @@ -0,0 +1,391 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use openshell_core::config::{ + DEFAULT_NETWORK_NAME, DEFAULT_SSH_HANDSHAKE_SKEW_SECS, DEFAULT_SSH_PORT, + DEFAULT_STOP_TIMEOUT_SECS, DEFAULT_SUPERVISOR_IMAGE, +}; +use std::path::PathBuf; +use std::str::FromStr; + +/// Image pull policy for sandbox and supervisor images. +/// +/// Controls when the Podman driver fetches a newer copy of an OCI image +/// from the registry. +#[derive(Debug, Clone, Default, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ImagePullPolicy { + /// Always pull, even if a local copy exists. + Always, + /// Pull only when no local copy exists (default). + #[default] + Missing, + /// Never pull; fail if not available locally. + Never, + /// Pull only if the remote image is newer. + Newer, +} + +impl ImagePullPolicy { + /// Return the policy string expected by the Podman libpod API. + #[must_use] + pub fn as_str(&self) -> &'static str { + match self { + Self::Always => "always", + Self::Missing => "missing", + Self::Never => "never", + Self::Newer => "newer", + } + } +} + +impl std::fmt::Display for ImagePullPolicy { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.as_str()) + } +} + +impl FromStr for ImagePullPolicy { + type Err = String; + + fn from_str(s: &str) -> Result { + match s.to_ascii_lowercase().as_str() { + "always" => Ok(Self::Always), + "missing" => Ok(Self::Missing), + "never" => Ok(Self::Never), + "newer" => Ok(Self::Newer), + other => Err(format!( + "invalid pull policy '{other}'; expected one of: always, missing, never, newer" + )), + } + } +} + +#[derive(Clone)] +pub struct PodmanComputeConfig { + /// Path to the Podman API Unix socket. + /// Default: `$XDG_RUNTIME_DIR/podman/podman.sock` (Linux), + /// `$HOME/.local/share/containers/podman/machine/podman.sock` (macOS). + pub socket_path: PathBuf, + /// Default OCI image for sandboxes. + pub default_image: String, + /// Image pull policy for sandbox images. + pub image_pull_policy: ImagePullPolicy, + /// Gateway gRPC endpoint the sandbox connects back to. + /// + /// When empty, the driver auto-detects the endpoint using + /// `gateway_port` and `host.containers.internal`. + pub grpc_endpoint: String, + /// Port the gateway server is actually listening on. + /// + /// Used by the driver's auto-detection fallback when `grpc_endpoint` + /// is empty. The server must set this to `config.bind_address.port()` + /// so the correct port is used even when `--port` differs from the + /// default. Defaults to [`openshell_core::config::DEFAULT_SERVER_PORT`]. + pub gateway_port: u16, + /// Unix socket path the in-container supervisor bridges relay traffic to. + pub sandbox_ssh_socket_path: String, + /// Name of the Podman bridge network. + /// Created automatically if it does not exist. + pub network_name: String, + /// SSH port inside the container. + pub ssh_port: u16, + /// Shared secret for the NSSH1 SSH handshake. + pub ssh_handshake_secret: String, + /// Maximum clock skew in seconds for SSH handshake timestamps. + pub ssh_handshake_skew_secs: u64, + /// Container stop timeout in seconds (SIGTERM → SIGKILL). + pub stop_timeout_secs: u32, + /// OCI image containing the openshell-sandbox supervisor binary. + /// Mounted read-only into sandbox containers at /opt/openshell/bin + /// using Podman's `type=image` mount. + pub supervisor_image: String, + /// Host path to the CA certificate for sandbox mTLS. + /// + /// When all three TLS paths (`guest_tls_ca`, `guest_tls_cert`, + /// `guest_tls_key`) are set, the driver bind-mounts them into sandbox + /// containers and switches the auto-detected endpoint from `http://` + /// to `https://`. + pub guest_tls_ca: Option, + /// Host path to the client certificate for sandbox mTLS. + pub guest_tls_cert: Option, + /// Host path to the client private key for sandbox mTLS. + pub guest_tls_key: Option, +} + +impl PodmanComputeConfig { + /// Returns `true` when all three TLS paths are configured. + #[must_use] + pub fn tls_enabled(&self) -> bool { + self.guest_tls_ca.is_some() && self.guest_tls_cert.is_some() && self.guest_tls_key.is_some() + } + + /// Validate TLS configuration consistency. + /// + /// Returns `Ok(())` when either all three TLS paths are set (full mTLS) + /// or none are set (plaintext). Returns an error naming the missing + /// fields when only a subset is provided — this prevents silent + /// fallback to plaintext when an operator partially configures mTLS. + pub fn validate_tls_config(&self) -> Result<(), crate::client::PodmanApiError> { + let has_ca = self.guest_tls_ca.is_some(); + let has_cert = self.guest_tls_cert.is_some(); + let has_key = self.guest_tls_key.is_some(); + + // All set or none set — both are valid. + if (has_ca && has_cert && has_key) || (!has_ca && !has_cert && !has_key) { + return Ok(()); + } + + let mut missing = Vec::new(); + if !has_ca { + missing.push("--podman-tls-ca / OPENSHELL_PODMAN_TLS_CA"); + } + if !has_cert { + missing.push("--podman-tls-cert / OPENSHELL_PODMAN_TLS_CERT"); + } + if !has_key { + missing.push("--podman-tls-key / OPENSHELL_PODMAN_TLS_KEY"); + } + + Err(crate::client::PodmanApiError::InvalidInput(format!( + "Partial TLS configuration: all three TLS paths must be provided together. \ + Missing: {}", + missing.join(", ") + ))) + } + + /// Resolve the default socket path from the environment. + /// + /// - **macOS**: `$HOME/.local/share/containers/podman/machine/podman.sock` + /// (the symlink created by `podman machine` pointing to the VM API socket). + /// - **Linux**: `$XDG_RUNTIME_DIR/podman/podman.sock` when set (by + /// `pam_systemd`/logind), otherwise `/run/user/{uid}/podman/podman.sock` + /// using the real UID via `getuid()`. + #[must_use] + pub fn default_socket_path() -> PathBuf { + #[cfg(target_os = "macos")] + { + let home = std::env::var("HOME").expect("HOME must be set on macOS"); + PathBuf::from(home).join(".local/share/containers/podman/machine/podman.sock") + } + #[cfg(target_os = "linux")] + { + std::env::var("XDG_RUNTIME_DIR").map_or_else( + |_| { + let uid = nix::unistd::getuid(); + PathBuf::from(format!("/run/user/{uid}/podman/podman.sock")) + }, + |xdg| PathBuf::from(xdg).join("podman/podman.sock"), + ) + } + } +} + +impl Default for PodmanComputeConfig { + fn default() -> Self { + Self { + socket_path: Self::default_socket_path(), + default_image: String::new(), + image_pull_policy: ImagePullPolicy::default(), + grpc_endpoint: String::new(), + gateway_port: openshell_core::config::DEFAULT_SERVER_PORT, + sandbox_ssh_socket_path: "/run/openshell/ssh.sock".to_string(), + network_name: DEFAULT_NETWORK_NAME.to_string(), + ssh_port: DEFAULT_SSH_PORT, + ssh_handshake_secret: String::new(), + ssh_handshake_skew_secs: DEFAULT_SSH_HANDSHAKE_SKEW_SECS, + stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS, + supervisor_image: DEFAULT_SUPERVISOR_IMAGE.to_string(), + guest_tls_ca: None, + guest_tls_cert: None, + guest_tls_key: None, + } + } +} + +impl std::fmt::Debug for PodmanComputeConfig { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PodmanComputeConfig") + .field("socket_path", &self.socket_path) + .field("default_image", &self.default_image) + .field("image_pull_policy", &self.image_pull_policy.as_str()) + .field("grpc_endpoint", &self.grpc_endpoint) + .field("gateway_port", &self.gateway_port) + .field("sandbox_ssh_socket_path", &self.sandbox_ssh_socket_path) + .field("network_name", &self.network_name) + .field("ssh_port", &self.ssh_port) + .field("ssh_handshake_secret", &"[REDACTED]") + .field("ssh_handshake_skew_secs", &self.ssh_handshake_skew_secs) + .field("stop_timeout_secs", &self.stop_timeout_secs) + .field("supervisor_image", &self.supervisor_image) + .field("guest_tls_ca", &self.guest_tls_ca) + .field("guest_tls_cert", &self.guest_tls_cert) + .field("guest_tls_key", &self.guest_tls_key) + .finish() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Serialises env-mutating tests so that parallel test threads cannot + /// observe each other's changes to `XDG_RUNTIME_DIR`. + static ENV_LOCK: std::sync::LazyLock> = + std::sync::LazyLock::new(|| std::sync::Mutex::new(())); + + #[test] + #[cfg(target_os = "linux")] + fn default_socket_path_respects_xdg_runtime_dir() { + let _guard = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + temp_env::with_vars([("XDG_RUNTIME_DIR", Some("/tmp/test-xdg"))], || { + let path = PodmanComputeConfig::default_socket_path(); + assert_eq!(path, PathBuf::from("/tmp/test-xdg/podman/podman.sock")); + }); + } + + #[test] + #[cfg(target_os = "linux")] + fn default_socket_path_falls_back_to_uid() { + let _guard = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + temp_env::with_vars([("XDG_RUNTIME_DIR", None::<&str>)], || { + let path = PodmanComputeConfig::default_socket_path(); + let uid = nix::unistd::getuid(); + assert_eq!( + path, + PathBuf::from(format!("/run/user/{uid}/podman/podman.sock")) + ); + }); + } + + #[test] + #[cfg(target_os = "macos")] + fn default_socket_path_uses_podman_machine_on_macos() { + let _guard = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + temp_env::with_vars([("HOME", Some("/Users/testuser"))], || { + let path = PodmanComputeConfig::default_socket_path(); + assert_eq!( + path, + PathBuf::from("/Users/testuser/.local/share/containers/podman/machine/podman.sock") + ); + }); + } + + // ── TLS config validation ───────────────────────────────────────── + + #[test] + fn validate_tls_config_all_none_is_ok() { + let cfg = PodmanComputeConfig::default(); + assert!(cfg.validate_tls_config().is_ok()); + } + + #[test] + fn validate_tls_config_all_set_is_ok() { + let cfg = PodmanComputeConfig { + guest_tls_ca: Some(PathBuf::from("/tls/ca.crt")), + guest_tls_cert: Some(PathBuf::from("/tls/tls.crt")), + guest_tls_key: Some(PathBuf::from("/tls/tls.key")), + ..PodmanComputeConfig::default() + }; + assert!(cfg.validate_tls_config().is_ok()); + } + + #[test] + fn validate_tls_config_only_ca_is_error() { + let cfg = PodmanComputeConfig { + guest_tls_ca: Some(PathBuf::from("/tls/ca.crt")), + ..PodmanComputeConfig::default() + }; + let err = cfg + .validate_tls_config() + .expect_err("only CA should be rejected"); + let msg = err.to_string(); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_CERT"), "{msg}"); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_KEY"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_CA"), "{msg}"); + } + + #[test] + fn validate_tls_config_only_cert_is_error() { + let cfg = PodmanComputeConfig { + guest_tls_cert: Some(PathBuf::from("/tls/tls.crt")), + ..PodmanComputeConfig::default() + }; + let err = cfg + .validate_tls_config() + .expect_err("only cert should be rejected"); + let msg = err.to_string(); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_CA"), "{msg}"); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_KEY"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_CERT"), "{msg}"); + } + + #[test] + fn validate_tls_config_only_key_is_error() { + let cfg = PodmanComputeConfig { + guest_tls_key: Some(PathBuf::from("/tls/tls.key")), + ..PodmanComputeConfig::default() + }; + let err = cfg + .validate_tls_config() + .expect_err("only key should be rejected"); + let msg = err.to_string(); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_CA"), "{msg}"); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_CERT"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_KEY"), "{msg}"); + } + + #[test] + fn validate_tls_config_ca_and_cert_missing_key_is_error() { + let cfg = PodmanComputeConfig { + guest_tls_ca: Some(PathBuf::from("/tls/ca.crt")), + guest_tls_cert: Some(PathBuf::from("/tls/tls.crt")), + ..PodmanComputeConfig::default() + }; + let err = cfg + .validate_tls_config() + .expect_err("missing key should be rejected"); + let msg = err.to_string(); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_KEY"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_CA"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_CERT"), "{msg}"); + } + + #[test] + fn validate_tls_config_ca_and_key_missing_cert_is_error() { + let cfg = PodmanComputeConfig { + guest_tls_ca: Some(PathBuf::from("/tls/ca.crt")), + guest_tls_key: Some(PathBuf::from("/tls/tls.key")), + ..PodmanComputeConfig::default() + }; + let err = cfg + .validate_tls_config() + .expect_err("missing cert should be rejected"); + let msg = err.to_string(); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_CERT"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_CA"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_KEY"), "{msg}"); + } + + #[test] + fn validate_tls_config_cert_and_key_missing_ca_is_error() { + let cfg = PodmanComputeConfig { + guest_tls_cert: Some(PathBuf::from("/tls/tls.crt")), + guest_tls_key: Some(PathBuf::from("/tls/tls.key")), + ..PodmanComputeConfig::default() + }; + let err = cfg + .validate_tls_config() + .expect_err("missing CA should be rejected"); + let msg = err.to_string(); + assert!(msg.contains("OPENSHELL_PODMAN_TLS_CA"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_CERT"), "{msg}"); + assert!(!msg.contains("OPENSHELL_PODMAN_TLS_KEY"), "{msg}"); + } +} diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs new file mode 100644 index 0000000000..ff0b68a46a --- /dev/null +++ b/crates/openshell-driver-podman/src/container.rs @@ -0,0 +1,1045 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Container spec construction for the Podman driver. + +use crate::config::PodmanComputeConfig; +use openshell_core::config::CDI_GPU_DEVICE_ALL; +use openshell_core::proto::compute::v1::DriverSandbox; +use serde::Serialize; +use serde_json::Value; +use std::collections::BTreeMap; + +/// Returns `true` when `SELinux` is enabled (enforcing or permissive). +/// +/// Checks whether selinuxfs is mounted, matching Podman's own detection +/// logic. Bind-mount relabeling (the `z` mount option) is needed in both +/// enforcing and permissive modes: enforcing blocks access outright, while +/// permissive floods the audit log with AVC denials that mask real issues. +/// +/// On non-`SELinux` systems (Ubuntu, macOS, Alpine) the directory does not +/// exist and this returns `false`, leaving mount options unchanged. +#[cfg(target_os = "linux")] +fn is_selinux_enabled() -> bool { + std::path::Path::new("/sys/fs/selinux").is_dir() +} + +#[cfg(not(target_os = "linux"))] +fn is_selinux_enabled() -> bool { + false +} + +/// Label key for the sandbox ID. +pub const LABEL_SANDBOX_ID: &str = "openshell.sandbox-id"; +/// Label key for the sandbox name. +pub const LABEL_SANDBOX_NAME: &str = "openshell.sandbox-name"; +/// Label applied to all managed containers. +pub const LABEL_MANAGED: &str = "openshell.managed"; +/// Label filter string for list/event queries. +pub const LABEL_MANAGED_FILTER: &str = "openshell.managed=true"; + +/// Container name prefix to avoid collisions with user containers. +const CONTAINER_PREFIX: &str = "openshell-sandbox-"; + +/// Volume name prefix. +const VOLUME_PREFIX: &str = "openshell-sandbox-"; + +/// Container-side mount paths for client TLS materials. +const TLS_CA_MOUNT_PATH: &str = "/etc/openshell/tls/client/ca.crt"; +const TLS_CERT_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.crt"; +const TLS_KEY_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.key"; + +/// Build a Podman container name from the sandbox name. +#[must_use] +pub fn container_name(sandbox_name: &str) -> String { + format!("{CONTAINER_PREFIX}{sandbox_name}") +} + +/// Build the workspace volume name from the sandbox ID. +#[must_use] +pub fn volume_name(sandbox_id: &str) -> String { + format!("{VOLUME_PREFIX}{sandbox_id}-workspace") +} + +/// Podman secret name prefix. +const SECRET_PREFIX: &str = "openshell-handshake-"; + +/// Build the Podman secret name for a sandbox's SSH handshake secret. +#[must_use] +pub fn secret_name(sandbox_id: &str) -> String { + format!("{SECRET_PREFIX}{sandbox_id}") +} + +/// Truncate a container ID to 12 characters (standard short form). +#[must_use] +pub fn short_id(id: &str) -> String { + id.chars().take(12).collect() +} + +// --------------------------------------------------------------------------- +// Typed container spec structs for the Podman libpod create API. +// --------------------------------------------------------------------------- + +#[derive(Serialize)] +struct ContainerSpec { + name: String, + image: String, + labels: BTreeMap, + env: BTreeMap, + volumes: Vec, + image_volumes: Vec, + hostname: String, + /// Overrides the image's ENTRYPOINT. In Podman's libpod API, `command` + /// only overrides CMD (appended as args to the entrypoint). We must set + /// `entrypoint` explicitly so the supervisor binary runs directly, + /// regardless of what ENTRYPOINT the sandbox image defines. + entrypoint: Vec, + command: Vec, + user: String, + cap_drop: Vec, + cap_add: Vec, + no_new_privileges: bool, + seccomp_profile_path: String, + image_pull_policy: String, + healthconfig: HealthConfig, + resource_limits: ResourceLimits, + /// Env-type secrets: map of `ENV_VAR_NAME → secret_name`. + /// Podman's libpod `SpecGenerator` uses `secret_env` (a flat map) for + /// environment-variable injection, distinct from `secrets` which only + /// handles file-mounted secrets under `/run/secrets/`. + secret_env: BTreeMap, + stop_timeout: u32, + /// Extra /etc/hosts entries. Used to inject `host.containers.internal` + /// via Podman's `host-gateway` magic so sandbox containers can reach + /// the gateway server running on the host in rootless mode. + hostadd: Vec, + netns: NetNS, + // Matches libpod's network spec format, which is `{name: {opts}}` where + // empty opts is a unit struct rather than `()`. Keep as a map so JSON + // serialization matches the API exactly. + #[allow(clippy::zero_sized_map_values)] + networks: BTreeMap, + #[serde(skip_serializing_if = "Option::is_none")] + devices: Option>, + /// Extra mounts for the libpod `SpecGenerator` (e.g. tmpfs entries). + mounts: Vec, + /// Port mappings from host to container. Using `host_port=0` requests an + /// ephemeral port, readable back from the inspect response. + portmappings: Vec, +} + +/// A port mapping entry for the libpod `SpecGenerator`. +#[derive(Serialize)] +struct PortMapping { + host_port: u16, + container_port: u16, + protocol: String, +} + +/// A mount entry for the libpod container create API `mounts` field. +/// +/// Unlike `volumes` (named Podman volumes) or `image_volumes` (OCI image +/// mounts resolved at the libpod layer), these mounts are passed to the +/// libpod `SpecGenerator` and support arbitrary mount types (e.g. tmpfs). +/// Field names must be lowercase to match the libpod JSON schema. +#[derive(Serialize)] +struct Mount { + #[serde(rename = "type")] + kind: String, + source: String, + destination: String, + #[serde(skip_serializing_if = "Vec::is_empty")] + options: Vec, +} + +/// A Podman image volume for the libpod container create API. +/// +/// Image volumes mount an OCI image's filesystem into a container without +/// running it. Podman resolves these at the libpod layer before generating +/// the OCI runtime spec, unlike `mounts` which are passed directly to the +/// OCI runtime (crun/runc). +#[derive(Serialize)] +struct ImageVolume { + source: String, + destination: String, + rw: bool, +} + +#[derive(Serialize)] +struct NamedVolume { + name: String, + dest: String, + options: Vec, +} + +#[derive(Serialize)] +struct HealthConfig { + test: Vec, + #[serde(rename = "Interval")] + interval: u64, + #[serde(rename = "Timeout")] + timeout: u64, + #[serde(rename = "Retries")] + retries: u32, + #[serde(rename = "StartPeriod")] + start_period: u64, +} + +#[derive(Serialize)] +struct ResourceLimits { + cpu: CpuLimits, + memory: MemoryLimits, +} + +#[derive(Serialize)] +struct CpuLimits { + quota: u64, + period: u64, +} + +#[derive(Serialize)] +struct MemoryLimits { + limit: u64, +} + +#[derive(Serialize)] +struct NetNS { + nsmode: String, +} + +#[derive(Serialize)] +struct NetworkAttachment {} + +#[derive(Serialize)] +struct LinuxDevice { + path: String, +} + +/// Default limits: 2 CPU cores (200000µs quota / 100000µs period), 4 GiB memory. +const DEFAULT_CPU_QUOTA: u64 = 200_000; +const DEFAULT_CPU_PERIOD: u64 = 100_000; +const DEFAULT_MEMORY_LIMIT: u64 = 4_294_967_296; // 4 GiB + +/// Resolve the OCI image reference for a sandbox, using the template image +/// if provided, otherwise the driver's default image. +#[must_use] +pub fn resolve_image<'a>(sandbox: &'a DriverSandbox, config: &'a PodmanComputeConfig) -> &'a str { + let spec = sandbox.spec.as_ref(); + let template = spec.and_then(|s| s.template.as_ref()); + template + .map(|t| t.image.as_str()) + .filter(|s| !s.is_empty()) + .unwrap_or(&config.default_image) +} + +/// Merge environment variables from user spec/template with required driver vars. +/// +/// User-supplied vars are inserted first so that the required driver +/// vars always win -- preventing spec/template overrides of security- +/// critical values like `OPENSHELL_ENDPOINT` or `OPENSHELL_SANDBOX_ID`. +fn build_env( + sandbox: &DriverSandbox, + config: &PodmanComputeConfig, + image: &str, +) -> BTreeMap { + let spec = sandbox.spec.as_ref(); + let template = spec.and_then(|s| s.template.as_ref()); + + let mut env: BTreeMap = BTreeMap::new(); + + // 1. User-supplied environment (lowest priority). + if let Some(s) = spec { + if !s.log_level.is_empty() { + env.insert("OPENSHELL_LOG_LEVEL".into(), s.log_level.clone()); + } + for (k, v) in &s.environment { + env.insert(k.clone(), v.clone()); + } + } + if let Some(t) = template { + for (k, v) in &t.environment { + env.insert(k.clone(), v.clone()); + } + } + + // 2. Required driver vars (highest priority -- always overwrite). + env.insert("OPENSHELL_SANDBOX".into(), sandbox.name.clone()); + env.insert("OPENSHELL_SANDBOX_ID".into(), sandbox.id.clone()); + env.insert("OPENSHELL_ENDPOINT".into(), config.grpc_endpoint.clone()); + env.insert( + "OPENSHELL_SSH_SOCKET_PATH".into(), + config.sandbox_ssh_socket_path.clone(), + ); + // NOTE: The SSH handshake secret is injected via a Podman secret + // (see the "secrets" field below) rather than a plaintext env var. + // This prevents exposure through `podman inspect`. + env.insert( + "OPENSHELL_SSH_HANDSHAKE_SKEW_SECS".into(), + config.ssh_handshake_skew_secs.to_string(), + ); + env.insert("OPENSHELL_CONTAINER_IMAGE".into(), image.to_string()); + env.insert("OPENSHELL_SANDBOX_COMMAND".into(), "sleep infinity".into()); + + // 3. TLS client cert paths (when mTLS is enabled). These point to + // the container-side mount paths where the cert files are + // bind-mounted from the host. + if config.tls_enabled() { + env.insert("OPENSHELL_TLS_CA".into(), TLS_CA_MOUNT_PATH.into()); + env.insert("OPENSHELL_TLS_CERT".into(), TLS_CERT_MOUNT_PATH.into()); + env.insert("OPENSHELL_TLS_KEY".into(), TLS_KEY_MOUNT_PATH.into()); + } + + env +} + +/// Merge labels from the sandbox template with required managed labels. +/// +/// User-supplied labels are inserted first so that the managed labels +/// always win -- preventing template overrides of internal tracking labels. +fn build_labels(sandbox: &DriverSandbox) -> BTreeMap { + let template = sandbox.spec.as_ref().and_then(|s| s.template.as_ref()); + + let mut labels: BTreeMap = BTreeMap::new(); + if let Some(t) = template { + for (k, v) in &t.labels { + labels.insert(k.clone(), v.clone()); + } + } + // Managed labels (highest priority -- always overwrite). + labels.insert(LABEL_SANDBOX_ID.into(), sandbox.id.clone()); + labels.insert(LABEL_SANDBOX_NAME.into(), sandbox.name.clone()); + labels.insert(LABEL_MANAGED.into(), "true".into()); + + labels +} + +/// Parse resource limits from the sandbox template, falling back to defaults. +fn build_resource_limits(sandbox: &DriverSandbox) -> ResourceLimits { + let resources = sandbox + .spec + .as_ref() + .and_then(|s| s.template.as_ref()) + .and_then(|t| t.resources.as_ref()); + + let cpu_micros = resources + .filter(|r| !r.cpu_limit.is_empty()) + .and_then(|r| parse_cpu_to_microseconds(&r.cpu_limit)) + .unwrap_or(DEFAULT_CPU_QUOTA); + + let mem_bytes = resources + .filter(|r| !r.memory_limit.is_empty()) + .and_then(|r| parse_memory_to_bytes(&r.memory_limit)) + .unwrap_or(DEFAULT_MEMORY_LIMIT); + + ResourceLimits { + cpu: CpuLimits { + quota: cpu_micros, + period: DEFAULT_CPU_PERIOD, + }, + memory: MemoryLimits { limit: mem_bytes }, + } +} + +/// Build CDI GPU device list if GPU is requested. +fn build_devices(sandbox: &DriverSandbox) -> Option> { + if sandbox.spec.as_ref().is_some_and(|s| s.gpu) { + Some(vec![LinuxDevice { + path: CDI_GPU_DEVICE_ALL.into(), + }]) + } else { + None + } +} + +/// Build the Podman container creation JSON spec. +#[must_use] +pub fn build_container_spec(sandbox: &DriverSandbox, config: &PodmanComputeConfig) -> Value { + let image = resolve_image(sandbox, config); + let name = container_name(&sandbox.name); + let vol = volume_name(&sandbox.id); + + let env = build_env(sandbox, config, image); + let labels = build_labels(sandbox); + let resource_limits = build_resource_limits(sandbox); + let devices = build_devices(sandbox); + + // Network configuration -- always bridge mode. + // Matches libpod's network spec format `{name: {opts}}`; the unit-struct + // value mirrors empty opts in the JSON. + #[allow(clippy::zero_sized_map_values)] + let mut networks = BTreeMap::new(); + networks.insert(config.network_name.clone(), NetworkAttachment {}); + + let container_spec = ContainerSpec { + name, + image: image.to_string(), + labels, + env, + volumes: vec![NamedVolume { + name: vol, + dest: "/sandbox".into(), + options: vec!["rw".into()], + }], + // Side-load the supervisor binary from a standalone OCI image. + // Podman resolves image_volumes at the libpod layer, mounting the + // image's filesystem at the destination path without starting a + // container from it. The supervisor image is FROM scratch with just + // the binary at /openshell-sandbox, so it appears at + // /opt/openshell/bin/openshell-sandbox. + image_volumes: vec![ImageVolume { + source: config.supervisor_image.clone(), + destination: "/opt/openshell/bin".into(), + rw: false, + }], + hostname: format!("sandbox-{}", sandbox.name), + // Override the image's ENTRYPOINT so the supervisor binary runs + // directly. Sandbox images (e.g. the community base image) set + // ENTRYPOINT ["/bin/bash"], and Podman's `command` field only + // overrides CMD — which gets appended as args to the entrypoint. + // Without this, the container would run `/bin/bash /opt/openshell/bin/openshell-sandbox` + // and bash would fail trying to interpret the binary as a script. + entrypoint: vec!["/opt/openshell/bin/openshell-sandbox".into()], + command: vec![], + // Force the supervisor to run as root (UID 0). Sandbox images may + // set a non-root USER directive (e.g. `USER sandbox`), but the + // supervisor needs root to create network namespaces, set up the + // proxy, and configure Landlock/seccomp. This matches the K8s + // driver's runAsUser: 0. + user: "0:0".into(), + // Podman's default container capability set is already restricted: + // CHOWN DAC_OVERRIDE FOWNER FSETID KILL SETGID SETUID SETPCAP + // NET_BIND_SERVICE SYS_CHROOT SETFCAP + // We add what the supervisor needs and drop what it doesn't. + cap_drop: vec![ + // Not needed: standard file permission bits are sufficient; dropping + // prevents the supervisor from bypassing DAC checks it shouldn't need. + "DAC_OVERRIDE".into(), + // Not needed: the supervisor does not create setuid/setgid executables. + "FSETID".into(), + // Not needed: the supervisor does not send signals to arbitrary processes. + "KILL".into(), + // Not needed: the supervisor does not bind privileged ports (<1024). + "NET_BIND_SERVICE".into(), + // Not in Podman's default set but explicitly denied in case the image + // or runtime adds it; raw sockets are not required. + "NET_RAW".into(), + // Not needed: the supervisor does not manipulate file capabilities. + "SETFCAP".into(), + // Not needed: the supervisor does not manage its own capability bounding set. + "SETPCAP".into(), + // Not needed: the supervisor does not call chroot(). + "SYS_CHROOT".into(), + ], + cap_add: vec![ + // seccomp filter installation, namespace creation, Landlock setup. + "SYS_ADMIN".into(), + // Network namespace veth setup, IP/route configuration. + "NET_ADMIN".into(), + // Reading /proc//exe and ancestor walk for process identity in policy. + "SYS_PTRACE".into(), + // Reading /dev/kmsg for bypass-detection diagnostics. + "SYSLOG".into(), + // Reading /proc//fd/ across UIDs for process identity resolution. + // In rootless Podman the supervisor runs as UID 0 inside a user namespace + // while sandbox processes run as the sandbox user. The kernel's + // proc_fd_permission() calls generic_permission() which denies cross-UID + // access to the dr-x------ fd directory unless this cap is present. + // Without it the proxy cannot determine which binary made each outbound + // connection and all traffic is denied. + "DAC_READ_SEARCH".into(), + ], + // SETUID, SETGID, CHOWN, and FOWNER are intentionally kept from Podman's + // default set and not dropped: + // SETUID/SETGID – drop_privileges(): setuid()/setgid()/initgroups() to the + // sandbox user. In rootless Podman cap_drop:ALL removes them + // from the bounding set even though uid=0 owns the user + // namespace — so we keep them by not dropping them explicitly. + // CHOWN – prepare_filesystem(): chown(path, uid, gid) on newly + // created read_write directories so the sandbox user can + // write to them. + // FOWNER – chown on files where the supervisor is not the owner + // (e.g. pre-existing directories owned by another user). + // + // Disable the container-level seccomp profile. The sandbox supervisor The sandbox supervisor + // installs its own policy-aware BPF seccomp filter at runtime via + // seccompiler (two-phase: clone3 blocker + main filter). The runtime + // filter is more restrictive than Podman's default — it blocks 20+ + // dangerous syscalls and conditionally restricts socket domains based + // on network policy. The filter self-seals by blocking further + // seccomp(SET_MODE_FILTER) calls after installation. + // + // A container-level profile would interfere by blocking the landlock + // and seccomp syscalls the supervisor needs during setup, before it + // locks itself down. + no_new_privileges: true, + seccomp_profile_path: "unconfined".into(), + image_pull_policy: config.image_pull_policy.as_str().to_string(), + healthconfig: HealthConfig { + test: vec![ + "CMD-SHELL".into(), + format!( + "test -e /var/run/openshell-ssh-ready || test -S {} || ss -tlnp | grep -q :{}", + config.sandbox_ssh_socket_path, config.ssh_port + ), + ], + interval: 3_000_000_000, + timeout: 2_000_000_000, + retries: 10, + start_period: 5_000_000_000, + }, + resource_limits, + // Inject the SSH handshake secret via Podman's secret_env map so it + // does not appear in `podman inspect` output. The libpod SpecGenerator + // uses `secret_env` (map of env_var → secret_name) for env-type secrets, + // distinct from `secrets` which only handles file mounts under /run/secrets/. + // The secret is created by the driver before the container + // (see `PodmanComputeDriver::create_sandbox`). + secret_env: BTreeMap::from([( + "OPENSHELL_SSH_HANDSHAKE_SECRET".into(), + secret_name(&sandbox.id), + )]), + stop_timeout: config.stop_timeout_secs, + // Inject host.containers.internal into /etc/hosts so sandbox + // containers can reach the gateway server on the host. The + // "host-gateway" magic value tells Podman to resolve to the + // host's actual IP (pasta uses 169.254.1.2 in rootless mode). + // This is the Podman equivalent of Docker's host.docker.internal. + hostadd: vec!["host.containers.internal:host-gateway".into()], + netns: NetNS { + nsmode: "bridge".to_string(), + }, + networks, + devices, + // Mount a tmpfs at /run/netns so the sandbox supervisor can create + // named network namespaces via `ip netns add`. The `ip` command requires + // /run/netns to exist and be bind-mountable; in rootless Podman this + // directory does not exist on the host, so the mkdir inside the container + // fails with EPERM. A private tmpfs gives the supervisor its own writable + // /run/netns without needing host filesystem access. + mounts: { + let mut m = vec![Mount { + kind: "tmpfs".into(), + source: "tmpfs".into(), + destination: "/run/netns".into(), + options: vec!["rw".into(), "nosuid".into(), "nodev".into()], + }]; + // Bind-mount client TLS materials into the container when mTLS + // is enabled. The supervisor reads these via OPENSHELL_TLS_CA, + // OPENSHELL_TLS_CERT, and OPENSHELL_TLS_KEY env vars (set in + // build_env above) to establish an mTLS connection back to the + // gateway. + if let (Some(ca), Some(cert), Some(key)) = ( + &config.guest_tls_ca, + &config.guest_tls_cert, + &config.guest_tls_key, + ) { + let mut ro = vec!["ro".into(), "rbind".into()]; + // On SELinux-enabled systems (Fedora, RHEL), bind-mounted + // files need the shared relabel option so the container + // process can read them through the SELinux MAC policy. + if is_selinux_enabled() { + ro.push("z".into()); + } + m.push(Mount { + kind: "bind".into(), + source: ca.display().to_string(), + destination: TLS_CA_MOUNT_PATH.into(), + options: ro.clone(), + }); + m.push(Mount { + kind: "bind".into(), + source: cert.display().to_string(), + destination: TLS_CERT_MOUNT_PATH.into(), + options: ro.clone(), + }); + m.push(Mount { + kind: "bind".into(), + source: key.display().to_string(), + destination: TLS_KEY_MOUNT_PATH.into(), + options: ro, + }); + } + m + }, + // Publish the SSH port with host_port=0 to get an ephemeral host port. + // In rootless Podman the bridge network (10.89.x.x) is not routable from + // the host, so we must use the published host port on 127.0.0.1 instead. + portmappings: vec![PortMapping { + host_port: 0, + container_port: config.ssh_port, + protocol: "tcp".into(), + }], + }; + + serde_json::to_value(container_spec).expect("ContainerSpec serialization cannot fail") +} + +/// Parse a Kubernetes-style CPU quantity to cgroup quota microseconds +/// (for a 100ms period). +/// +/// Examples: `"500m"` → 50000, `"2"` → 200000, `"0.5"` → 50000. +fn parse_cpu_to_microseconds(quantity: &str) -> Option { + let micros = if let Some(millis_str) = quantity.strip_suffix('m') { + let millis: u64 = millis_str.parse().ok()?; + // quota = millis * period / 1000 + millis.checked_mul(100)? + } else { + let cores: f64 = quantity.parse().ok()?; + if cores <= 0.0 || cores.is_nan() || cores.is_infinite() { + return None; + } + #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] + let val = (cores * 100_000.0) as u64; + val + }; + // A quota of 0 microseconds is invalid — treat as no limit. + if micros == 0 { None } else { Some(micros) } +} + +/// Parse a Kubernetes-style memory quantity to bytes. +/// +/// Supports: `Ki`, `Mi`, `Gi`, `Ti` (binary) and `k`, `M`, `G`, `T` +/// (decimal), as well as plain byte values. +fn parse_memory_to_bytes(quantity: &str) -> Option { + let suffixes: &[(&str, u64)] = &[ + ("Ti", 1024 * 1024 * 1024 * 1024), + ("Gi", 1024 * 1024 * 1024), + ("Mi", 1024 * 1024), + ("Ki", 1024), + ("T", 1_000_000_000_000), + ("G", 1_000_000_000), + ("M", 1_000_000), + ("k", 1_000), + ]; + + for (suffix, multiplier) in suffixes { + if let Some(num_str) = quantity.strip_suffix(suffix) { + let num: u64 = num_str.parse().ok()?; + return num.checked_mul(*multiplier); + } + } + + // Plain bytes. + quantity.parse().ok() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_cpu_millicore() { + assert_eq!(parse_cpu_to_microseconds("500m"), Some(50_000)); + assert_eq!(parse_cpu_to_microseconds("1000m"), Some(100_000)); + assert_eq!(parse_cpu_to_microseconds("250m"), Some(25_000)); + } + + #[test] + fn parse_cpu_whole_cores() { + assert_eq!(parse_cpu_to_microseconds("1"), Some(100_000)); + assert_eq!(parse_cpu_to_microseconds("2"), Some(200_000)); + assert_eq!(parse_cpu_to_microseconds("0.5"), Some(50_000)); + } + + #[test] + fn parse_memory_binary_suffixes() { + assert_eq!(parse_memory_to_bytes("256Mi"), Some(256 * 1024 * 1024)); + assert_eq!(parse_memory_to_bytes("4Gi"), Some(4 * 1024 * 1024 * 1024)); + assert_eq!(parse_memory_to_bytes("1Ki"), Some(1024)); + } + + #[test] + fn parse_memory_decimal_suffixes() { + assert_eq!(parse_memory_to_bytes("1G"), Some(1_000_000_000)); + assert_eq!(parse_memory_to_bytes("500M"), Some(500_000_000)); + } + + #[test] + fn parse_memory_plain_bytes() { + assert_eq!(parse_memory_to_bytes("1048576"), Some(1_048_576)); + } + + #[test] + fn container_name_is_prefixed() { + assert_eq!(container_name("my-sandbox"), "openshell-sandbox-my-sandbox"); + } + + #[test] + fn volume_name_uses_id() { + assert_eq!( + volume_name("abc-123"), + "openshell-sandbox-abc-123-workspace" + ); + } + + #[test] + fn secret_name_uses_id() { + assert_eq!(secret_name("abc-123"), "openshell-handshake-abc-123"); + } + + #[test] + fn short_id_truncates() { + assert_eq!(short_id("abc123def456789"), "abc123def456"); + assert_eq!(short_id("short"), "short"); + } + + #[test] + fn container_spec_includes_required_capabilities() { + let sandbox = test_sandbox("test-id", "test-name"); + let config = test_config(); + let spec = build_container_spec(&sandbox, &config); + + let added: Vec<&str> = spec["cap_add"] + .as_array() + .expect("cap_add should be an array") + .iter() + .filter_map(|v| v.as_str()) + .collect(); + assert!(added.contains(&"SYS_ADMIN"), "missing SYS_ADMIN"); + assert!(added.contains(&"NET_ADMIN"), "missing NET_ADMIN"); + assert!(added.contains(&"SYS_PTRACE"), "missing SYS_PTRACE"); + assert!(added.contains(&"SYSLOG"), "missing SYSLOG"); + assert!( + added.contains(&"DAC_READ_SEARCH"), + "missing DAC_READ_SEARCH" + ); + + // SETUID and SETGID are NOT in cap_add — they remain available from the + // default bounding set because we no longer use cap_drop:ALL. Verify they + // are also not explicitly dropped. Similarly CHOWN and FOWNER must not be + // dropped because prepare_filesystem() calls chown() on newly created + // read_write directories before the supervisor drops privileges. + let dropped: Vec<&str> = spec["cap_drop"] + .as_array() + .expect("cap_drop should be an array") + .iter() + .filter_map(|v| v.as_str()) + .collect(); + assert!(!dropped.contains(&"SETUID"), "SETUID must not be dropped"); + assert!(!dropped.contains(&"SETGID"), "SETGID must not be dropped"); + assert!( + !dropped.contains(&"CHOWN"), + "CHOWN must not be dropped (needed for prepare_filesystem chown)" + ); + assert!( + !dropped.contains(&"FOWNER"), + "FOWNER must not be dropped (needed for chown on non-owned files)" + ); + assert!( + !dropped.contains(&"ALL"), + "must not use cap_drop:ALL in rootless Podman" + ); + } + + #[test] + fn container_spec_uses_secret_env_not_plaintext() { + let sandbox = test_sandbox("test-id", "test-name"); + let config = test_config(); + let spec = build_container_spec(&sandbox, &config); + + // The handshake secret must NOT appear in the plaintext env map. + let env_map = spec["env"].as_object().expect("env should be an object"); + assert!( + !env_map.contains_key("OPENSHELL_SSH_HANDSHAKE_SECRET"), + "handshake secret should not be in plaintext env" + ); + + // It should appear in secret_env (the libpod env-type secret map) instead. + let secret_env = spec["secret_env"] + .as_object() + .expect("secret_env should be an object"); + assert!( + secret_env.contains_key("OPENSHELL_SSH_HANDSHAKE_SECRET"), + "secret_env should map OPENSHELL_SSH_HANDSHAKE_SECRET to its secret name" + ); + assert_eq!( + secret_env["OPENSHELL_SSH_HANDSHAKE_SECRET"].as_str(), + Some("openshell-handshake-test-id"), + "secret_env value should be the Podman secret name for the sandbox" + ); + } + + #[test] + fn container_spec_sets_sandbox_name_in_env() { + let sandbox = test_sandbox("test-id", "my-sandbox"); + let config = test_config(); + let spec = build_container_spec(&sandbox, &config); + + let env_map = spec["env"].as_object().expect("env should be an object"); + assert_eq!( + env_map.get("OPENSHELL_SANDBOX").and_then(|v| v.as_str()), + Some("my-sandbox"), + ); + } + + #[test] + fn container_spec_sets_ssh_socket_path_in_env() { + let sandbox = test_sandbox("test-id", "test-name"); + let config = test_config(); + let spec = build_container_spec(&sandbox, &config); + + let env_map = spec["env"].as_object().expect("env should be an object"); + assert_eq!( + env_map + .get("OPENSHELL_SSH_SOCKET_PATH") + .and_then(|v| v.as_str()), + Some("/run/openshell/test-ssh.sock"), + ); + } + + #[test] + fn container_spec_healthcheck_accepts_supervisor_socket() { + let sandbox = test_sandbox("test-id", "test-name"); + let config = test_config(); + let spec = build_container_spec(&sandbox, &config); + + let healthcheck = spec["healthconfig"]["test"] + .as_array() + .expect("healthcheck test should be an array"); + let command = healthcheck + .get(1) + .and_then(|v| v.as_str()) + .expect("healthcheck should include shell command"); + assert!( + command.contains("test -S /run/openshell/test-ssh.sock"), + "healthcheck should consider the supervisor Unix socket ready" + ); + } + + #[test] + fn container_spec_required_vars_cannot_be_overridden() { + use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate}; + + let mut sandbox = test_sandbox("test-id", "legit-name"); + let mut env_overrides = std::collections::HashMap::new(); + env_overrides.insert( + "OPENSHELL_ENDPOINT".to_string(), + "http://evil.example.com".to_string(), + ); + env_overrides.insert("OPENSHELL_SANDBOX_ID".to_string(), "spoofed-id".to_string()); + env_overrides.insert( + "OPENSHELL_SSH_SOCKET_PATH".to_string(), + "/tmp/evil.sock".to_string(), + ); + sandbox.spec = Some(DriverSandboxSpec { + environment: env_overrides, + template: Some(DriverSandboxTemplate::default()), + ..Default::default() + }); + + let config = test_config(); + let spec = build_container_spec(&sandbox, &config); + + let env_map = spec["env"].as_object().expect("env should be an object"); + + assert_eq!( + env_map.get("OPENSHELL_ENDPOINT").and_then(|v| v.as_str()), + Some("http://localhost:50051"), + "OPENSHELL_ENDPOINT must not be overridden by user env" + ); + assert_eq!( + env_map.get("OPENSHELL_SANDBOX_ID").and_then(|v| v.as_str()), + Some("test-id"), + "OPENSHELL_SANDBOX_ID must not be overridden by user env" + ); + assert_eq!( + env_map + .get("OPENSHELL_SSH_SOCKET_PATH") + .and_then(|v| v.as_str()), + Some("/run/openshell/test-ssh.sock"), + "OPENSHELL_SSH_SOCKET_PATH must not be overridden by user env" + ); + } + + #[test] + fn container_spec_required_labels_cannot_be_overridden() { + use openshell_core::proto::compute::v1::{DriverSandboxSpec, DriverSandboxTemplate}; + + let mut sandbox = test_sandbox("real-id", "real-name"); + let mut label_overrides = std::collections::HashMap::new(); + label_overrides.insert("openshell.sandbox-id".to_string(), "spoofed-id".to_string()); + label_overrides.insert( + "openshell.sandbox-name".to_string(), + "spoofed-name".to_string(), + ); + sandbox.spec = Some(DriverSandboxSpec { + template: Some(DriverSandboxTemplate { + labels: label_overrides, + ..Default::default() + }), + ..Default::default() + }); + + let config = test_config(); + let spec = build_container_spec(&sandbox, &config); + + let labels = spec["labels"] + .as_object() + .expect("labels should be an object"); + assert_eq!( + labels.get("openshell.sandbox-id").and_then(|v| v.as_str()), + Some("real-id"), + "openshell.sandbox-id must not be overridden by template labels" + ); + assert_eq!( + labels + .get("openshell.sandbox-name") + .and_then(|v| v.as_str()), + Some("real-name"), + "openshell.sandbox-name must not be overridden by template labels" + ); + } + + #[test] + fn parse_cpu_negative_returns_none() { + assert_eq!(parse_cpu_to_microseconds("-1"), None); + assert_eq!(parse_cpu_to_microseconds("-500m"), None); + } + + #[test] + fn parse_cpu_zero_returns_none() { + assert_eq!(parse_cpu_to_microseconds("0m"), None); + assert_eq!(parse_cpu_to_microseconds("0"), None); + } + + fn test_sandbox(id: &str, name: &str) -> DriverSandbox { + DriverSandbox { + id: id.to_string(), + name: name.to_string(), + namespace: String::new(), + spec: None, + status: None, + } + } + + fn test_config() -> PodmanComputeConfig { + PodmanComputeConfig { + socket_path: std::path::PathBuf::from("/tmp/test.sock"), + default_image: "test-image:latest".to_string(), + grpc_endpoint: "http://localhost:50051".to_string(), + sandbox_ssh_socket_path: "/run/openshell/test-ssh.sock".to_string(), + ssh_handshake_secret: "test-secret-value".to_string(), + ..PodmanComputeConfig::default() + } + } + + #[test] + fn container_spec_includes_supervisor_image_volume() { + let sandbox = test_sandbox("test-id", "test-name"); + let config = test_config(); + let spec = build_container_spec(&sandbox, &config); + + let image_volumes = spec["image_volumes"] + .as_array() + .expect("image_volumes should be an array"); + assert_eq!( + image_volumes.len(), + 1, + "should have exactly one image volume" + ); + + let vol = &image_volumes[0]; + assert_eq!( + vol["source"].as_str(), + Some("openshell/supervisor:latest"), + "image volume source should be the supervisor image" + ); + assert_eq!( + vol["destination"].as_str(), + Some("/opt/openshell/bin"), + "image volume destination should be /opt/openshell/bin" + ); + assert_eq!( + vol["rw"].as_bool(), + Some(false), + "image volume should be read-only" + ); + } + + #[test] + fn container_spec_includes_tls_mounts_when_configured() { + let sandbox = test_sandbox("tls-id", "tls-name"); + let mut config = test_config(); + config.guest_tls_ca = Some(std::path::PathBuf::from("/host/ca.crt")); + config.guest_tls_cert = Some(std::path::PathBuf::from("/host/tls.crt")); + config.guest_tls_key = Some(std::path::PathBuf::from("/host/tls.key")); + + let spec = build_container_spec(&sandbox, &config); + + // Verify TLS env vars are set. + let env_map = spec["env"].as_object().expect("env should be an object"); + assert_eq!( + env_map.get("OPENSHELL_TLS_CA").and_then(|v| v.as_str()), + Some("/etc/openshell/tls/client/ca.crt"), + ); + assert_eq!( + env_map.get("OPENSHELL_TLS_CERT").and_then(|v| v.as_str()), + Some("/etc/openshell/tls/client/tls.crt"), + ); + assert_eq!( + env_map.get("OPENSHELL_TLS_KEY").and_then(|v| v.as_str()), + Some("/etc/openshell/tls/client/tls.key"), + ); + + // Verify bind mounts exist for all three cert files. + let mounts = spec["mounts"] + .as_array() + .expect("mounts should be an array"); + let bind_dests: Vec<&str> = mounts + .iter() + .filter(|m| m["type"].as_str() == Some("bind")) + .filter_map(|m| m["destination"].as_str()) + .collect(); + assert!( + bind_dests.contains(&"/etc/openshell/tls/client/ca.crt"), + "should bind-mount CA cert" + ); + assert!( + bind_dests.contains(&"/etc/openshell/tls/client/tls.crt"), + "should bind-mount client cert" + ); + assert!( + bind_dests.contains(&"/etc/openshell/tls/client/tls.key"), + "should bind-mount client key" + ); + + // Verify SELinux relabel option is present iff SELinux is enabled. + let tls_binds: Vec<&Value> = mounts + .iter() + .filter(|m| m["type"].as_str() == Some("bind")) + .collect(); + let has_z = tls_binds.iter().all(|m| { + m["options"] + .as_array() + .is_some_and(|opts| opts.iter().any(|o| o.as_str() == Some("z"))) + }); + assert_eq!( + has_z, + is_selinux_enabled(), + "TLS bind mounts should include 'z' option iff SELinux is enabled" + ); + } + + #[test] + fn container_spec_omits_tls_without_config() { + let sandbox = test_sandbox("notls-id", "notls-name"); + let config = test_config(); + + let spec = build_container_spec(&sandbox, &config); + + let env_map = spec["env"].as_object().expect("env should be an object"); + assert!( + env_map.get("OPENSHELL_TLS_CA").is_none(), + "TLS env vars should not be set without TLS config" + ); + + let mounts = spec["mounts"] + .as_array() + .expect("mounts should be an array"); + let bind_count = mounts + .iter() + .filter(|m| m["type"].as_str() == Some("bind")) + .count(); + assert_eq!(bind_count, 0, "no bind mounts without TLS config"); + } +} diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs new file mode 100644 index 0000000000..ad4d7a192c --- /dev/null +++ b/crates/openshell-driver-podman/src/driver.rs @@ -0,0 +1,871 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Podman compute driver. + +use crate::client::{PodmanApiError, PodmanClient}; +use crate::config::PodmanComputeConfig; +use crate::container::{self, LABEL_MANAGED_FILTER, LABEL_SANDBOX_ID}; +use crate::watcher::{ + self, WatchStream, driver_sandbox_from_inspect, driver_sandbox_from_list_entry, +}; +use openshell_core::ComputeDriverError; +use openshell_core::proto::compute::v1::{DriverSandbox, GetCapabilitiesResponse}; +use tracing::{info, warn}; + +impl From for ComputeDriverError { + fn from(value: PodmanApiError) -> Self { + match value { + PodmanApiError::Conflict(_) => Self::AlreadyExists, + PodmanApiError::NotFound(msg) => Self::Message(format!("not found: {msg}")), + other => Self::Message(other.to_string()), + } + } +} + +/// Podman compute driver managing sandbox containers via the Podman REST API. +#[derive(Clone)] +pub struct PodmanComputeDriver { + client: PodmanClient, + config: PodmanComputeConfig, + /// The host's IP on the bridge network. Sandbox containers use this to + /// reach the gateway server when no explicit gRPC endpoint is configured. + network_gateway_ip: Option, +} + +impl std::fmt::Debug for PodmanComputeDriver { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PodmanComputeDriver") + .field("socket_path", &self.config.socket_path) + .field("default_image", &self.config.default_image) + .field("network_name", &self.config.network_name) + .finish() + } +} + +/// Construct and validate a container name from a sandbox name. +/// +/// Combines the prefix with the sandbox name and validates the result +/// against Podman's naming rules before any resources are created. +fn validated_container_name(sandbox_name: &str) -> Result { + let name = container::container_name(sandbox_name); + crate::client::validate_name(&name) + .map_err(|e| ComputeDriverError::Precondition(e.to_string()))?; + Ok(name) +} + +impl PodmanComputeDriver { + /// Create a new driver, verifying the Podman socket is reachable. + pub async fn new(mut config: PodmanComputeConfig) -> Result { + if !config.socket_path.exists() { + if cfg!(target_os = "macos") { + warn!( + path = %config.socket_path.display(), + "Podman socket not found; is podman machine running? \ + Try `podman machine start` or set OPENSHELL_PODMAN_SOCKET to override." + ); + } else { + warn!( + path = %config.socket_path.display(), + "Podman socket not found; is the Podman service running? \ + Set OPENSHELL_PODMAN_SOCKET or XDG_RUNTIME_DIR to override." + ); + } + } + + // Validate TLS configuration before connecting. Partial configs + // (e.g. CA set but cert/key missing) are rejected early so operators + // get a clear error instead of a silent fallback to plaintext HTTP. + config.validate_tls_config()?; + + let client = PodmanClient::new(config.socket_path.clone()); + + // Verify connectivity. + client.ping().await?; + + // Verify cgroups v2, detect rootless mode, and log system info. + match client.system_info().await { + Ok(info) => { + if info.host.cgroup_version != "v2" { + return Err(PodmanApiError::Connection(format!( + "cgroups v2 is required; detected cgroups '{}'. \ + Ensure your host uses a unified cgroup hierarchy \ + (systemd.unified_cgroup_hierarchy=1).", + info.host.cgroup_version + ))); + } + info!( + cgroup_version = %info.host.cgroup_version, + network_backend = %info.host.network_backend, + rootless = info.host.security.rootless, + "Connected to Podman" + ); + } + Err(e) => { + return Err(PodmanApiError::Connection(format!( + "failed to query Podman system info: {e}" + ))); + } + } + + // Rootless pre-flight: warn if subuid/subgid ranges look missing. + // Not a hard error because some systems configure these via LDAP or + // other mechanisms that /etc/subuid does not reflect. + if nix::unistd::getuid().as_raw() != 0 { + check_subuid_range(); + } + + // Ensure the bridge network exists. + client.ensure_network(&config.network_name).await?; + let network_gateway_ip = client + .network_gateway_ip(&config.network_name) + .await + .unwrap_or(None); + info!( + network = %config.network_name, + gateway_ip = ?network_gateway_ip, + "Bridge network ready" + ); + + // Auto-detect the gRPC callback endpoint when not explicitly + // configured. Sandbox containers use host.containers.internal + // (injected via hostadd with host-gateway in the container spec) + // to reach the gateway server on the host. The scheme is + // determined by whether TLS client certs are configured: when + // all three TLS paths are set, the endpoint uses https so the + // supervisor connects with mTLS. + if config.grpc_endpoint.is_empty() { + let scheme = if config.tls_enabled() { + "https" + } else { + "http" + }; + config.grpc_endpoint = format!( + "{scheme}://host.containers.internal:{}", + config.gateway_port + ); + info!( + grpc_endpoint = %config.grpc_endpoint, + tls = config.tls_enabled(), + "Auto-detected gRPC endpoint" + ); + } + + Ok(Self { + client, + config, + network_gateway_ip, + }) + } + + /// The host's IP on the bridge network, if available. + /// + /// Used by the server to auto-detect the gRPC callback endpoint when + /// no explicit `--grpc-endpoint` is configured. + #[must_use] + pub fn network_gateway_ip(&self) -> Option<&str> { + self.network_gateway_ip.as_deref() + } + + /// Report driver capabilities. + pub fn capabilities(&self) -> Result { + let supports_gpu = Self::has_gpu_capacity(); + Ok(GetCapabilitiesResponse { + driver_name: "podman".to_string(), + driver_version: openshell_core::VERSION.to_string(), + default_image: self.config.default_image.clone(), + supports_gpu, + gpu_count: 0, + }) + } + + #[must_use] + pub fn default_image(&self) -> &str { + &self.config.default_image + } + + /// Check whether GPU devices are available via CDI. + /// + /// The Podman system info response doesn't directly list CDI devices in all + /// versions. As a heuristic, check if the NVIDIA device node exists (this + /// works for both rootful and rootless). + fn has_gpu_capacity() -> bool { + std::path::Path::new("/dev/nvidia0").exists() + } + + /// Validate a sandbox before creation. + pub fn validate_sandbox_create( + &self, + sandbox: &DriverSandbox, + ) -> Result<(), ComputeDriverError> { + let gpu_requested = sandbox.spec.as_ref().is_some_and(|s| s.gpu); + if gpu_requested && !Self::has_gpu_capacity() { + return Err(ComputeDriverError::Precondition( + "GPU sandbox requested, but no NVIDIA GPU devices are available.".to_string(), + )); + } + Ok(()) + } + + /// Create a sandbox container. + pub async fn create_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), ComputeDriverError> { + if sandbox.name.is_empty() { + return Err(ComputeDriverError::Precondition( + "sandbox name is required".into(), + )); + } + if sandbox.id.is_empty() { + return Err(ComputeDriverError::Precondition( + "sandbox id is required".into(), + )); + } + + // Validate the composed container name early, before creating any + // resources (secret, volume), so we don't leave orphans when the + // name is invalid. + let name = validated_container_name(&sandbox.name)?; + + let vol_name = container::volume_name(&sandbox.id); + let sec_name = container::secret_name(&sandbox.id); + + info!( + sandbox_id = %sandbox.id, + sandbox_name = %sandbox.name, + container = %name, + "Creating sandbox container" + ); + + // 1a. Pull the supervisor image if needed. The supervisor binary + // is shipped in a standalone OCI image and mounted into sandbox + // containers via Podman's type=image mount. Using "missing" + // policy so the image is only pulled once and then cached. + info!( + image = %self.config.supervisor_image, + policy = "missing", + "Ensuring supervisor image" + ); + self.client + .pull_image(&self.config.supervisor_image, "missing") + .await + .map_err(ComputeDriverError::from)?; + + // 1b. Pull the sandbox image if needed (Podman does not pull on create). + let image = container::resolve_image(sandbox, &self.config); + if image.is_empty() { + return Err(ComputeDriverError::Precondition( + "no sandbox image configured: set --sandbox-image on the server \ + or provide an image in the sandbox template" + .to_string(), + )); + } + let pull_policy = self.config.image_pull_policy.as_str(); + info!(image = %image, policy = %pull_policy, "Ensuring sandbox image"); + self.client + .pull_image(image, pull_policy) + .await + .map_err(ComputeDriverError::from)?; + + // 2. Create the SSH handshake secret via the Podman secrets API + // so it is not exposed in `podman inspect` output. + self.client + .create_secret(&sec_name, self.config.ssh_handshake_secret.as_bytes()) + .await + .map_err(ComputeDriverError::from)?; + + // 3. Create workspace volume. + if let Err(e) = self.client.create_volume(&vol_name).await { + let _ = self.client.remove_secret(&sec_name).await; + return Err(ComputeDriverError::from(e)); + } + + // 4. Create container. + let spec = container::build_container_spec(sandbox, &self.config); + match self.client.create_container(&spec).await { + Ok(_) => {} + Err(PodmanApiError::Conflict(_)) => { + // Clean up the volume and secret we just created. They are + // keyed by *this* sandbox's ID, not the conflicting + // container's ID (which has the same name but a different + // ID), so they would be orphaned otherwise. + let _ = self.client.remove_volume(&vol_name).await; + let _ = self.client.remove_secret(&sec_name).await; + return Err(ComputeDriverError::AlreadyExists); + } + Err(e) => { + let _ = self.client.remove_volume(&vol_name).await; + let _ = self.client.remove_secret(&sec_name).await; + return Err(ComputeDriverError::from(e)); + } + } + + // 5. Start container. + if let Err(e) = self.client.start_container(&name).await { + warn!( + sandbox_name = %sandbox.name, + error = %e, + "Failed to start container; cleaning up" + ); + let _ = self.client.remove_container(&name).await; + let _ = self.client.remove_volume(&vol_name).await; + let _ = self.client.remove_secret(&sec_name).await; + return Err(ComputeDriverError::from(e)); + } + + info!( + sandbox_id = %sandbox.id, + sandbox_name = %sandbox.name, + "Sandbox container started" + ); + + Ok(()) + } + + /// Stop a sandbox container without deleting it. + pub async fn stop_sandbox(&self, sandbox_name: &str) -> Result<(), ComputeDriverError> { + let name = validated_container_name(sandbox_name)?; + info!(sandbox_name = %sandbox_name, container = %name, "Stopping sandbox container"); + + self.client + .stop_container(&name, self.config.stop_timeout_secs) + .await + .map_err(ComputeDriverError::from) + } + + /// Delete a sandbox container and its workspace volume. + pub async fn delete_sandbox( + &self, + sandbox_id: &str, + sandbox_name: &str, + ) -> Result { + if sandbox_id.is_empty() { + return Err(ComputeDriverError::Precondition( + "sandbox id is required".into(), + )); + } + let name = validated_container_name(sandbox_name)?; + info!( + sandbox_id = %sandbox_id, + sandbox_name = %sandbox_name, + container = %name, + "Deleting sandbox container" + ); + + // Use the request's stable sandbox ID as the source of truth for + // cleanup. Inspect is only used as a best-effort cross-check so + // cleanup still works if the container is already gone or mislabeled. + match self.client.inspect_container(&name).await { + Ok(inspect) => match inspect.config.labels.get(LABEL_SANDBOX_ID) { + Some(label_id) if label_id != sandbox_id => { + warn!( + sandbox_id = %sandbox_id, + sandbox_name = %sandbox_name, + container = %name, + label_sandbox_id = %label_id, + "Container label sandbox ID did not match delete request; cleaning up using request sandbox_id" + ); + } + None => { + warn!( + sandbox_id = %sandbox_id, + sandbox_name = %sandbox_name, + container = %name, + "Container missing '{}' label; cleaning up using request sandbox_id", + LABEL_SANDBOX_ID, + ); + } + Some(_) => {} + }, + Err(PodmanApiError::NotFound(_)) => {} + Err(e) => return Err(ComputeDriverError::from(e)), + } + + // Stop (best-effort). + let _ = self + .client + .stop_container(&name, self.config.stop_timeout_secs) + .await; + + // Remove container. If NotFound, the container was removed between + // inspect and here (TOCTOU race); proceed with volume/secret cleanup + // since those resources are idempotent to remove. + let container_existed = match self.client.remove_container(&name).await { + Ok(()) => true, + Err(PodmanApiError::NotFound(_)) => false, + Err(e) => return Err(ComputeDriverError::from(e)), + }; + + // Remove workspace volume and handshake secret. + let vol = container::volume_name(sandbox_id); + if let Err(e) = self.client.remove_volume(&vol).await { + warn!( + sandbox_id = %sandbox_id, + sandbox_name = %sandbox_name, + volume = %vol, + error = %e, + "Failed to remove workspace volume" + ); + } + let sec = container::secret_name(sandbox_id); + if let Err(e) = self.client.remove_secret(&sec).await { + warn!( + sandbox_id = %sandbox_id, + sandbox_name = %sandbox_name, + secret = %sec, + error = %e, + "Failed to remove handshake secret" + ); + } + + Ok(container_existed) + } + + /// Check whether a sandbox container exists. + pub async fn sandbox_exists(&self, sandbox_name: &str) -> Result { + let name = container::container_name(sandbox_name); + match self.client.inspect_container(&name).await { + Ok(_) => Ok(true), + Err(PodmanApiError::NotFound(_)) => Ok(false), + Err(e) => Err(ComputeDriverError::from(e)), + } + } + + /// Fetch a single sandbox by name. + pub async fn get_sandbox( + &self, + sandbox_name: &str, + ) -> Result, ComputeDriverError> { + let name = container::container_name(sandbox_name); + match self.client.inspect_container(&name).await { + Ok(inspect) => Ok(driver_sandbox_from_inspect(&inspect)), + Err(PodmanApiError::NotFound(_)) => Ok(None), + Err(e) => Err(ComputeDriverError::from(e)), + } + } + + /// List all managed sandboxes. + /// + /// Only inspects running containers (to get health status). Non-running + /// containers are built directly from the list entry data. + pub async fn list_sandboxes(&self) -> Result, ComputeDriverError> { + let entries = self + .client + .list_containers(LABEL_MANAGED_FILTER) + .await + .map_err(ComputeDriverError::from)?; + + let mut sandboxes = Vec::with_capacity(entries.len()); + for entry in &entries { + if entry.state == "running" { + // Running containers need inspect for health check status. + match self.client.inspect_container(&entry.id).await { + Ok(inspect) => { + if let Some(sandbox) = driver_sandbox_from_inspect(&inspect) { + sandboxes.push(sandbox); + continue; + } + } + Err(e) => { + let name = entry.names.first().cloned().unwrap_or_default(); + warn!( + container = %name, + error = %e, + "Failed to inspect running container during list, falling back to list entry" + ); + } + } + } + // Non-running containers (or inspect fallback): build from list data. + if let Some(sandbox) = driver_sandbox_from_list_entry(entry) { + sandboxes.push(sandbox); + } + } + + sandboxes.sort_by(|a, b| a.name.cmp(&b.name).then_with(|| a.id.cmp(&b.id))); + Ok(sandboxes) + } + + /// Start watching all managed sandbox containers. + pub async fn watch_sandboxes(&self) -> Result { + watcher::start_watch(self.client.clone()) + .await + .map_err(ComputeDriverError::from) + } +} + +#[cfg(test)] +impl PodmanComputeDriver { + pub(crate) fn for_tests(config: PodmanComputeConfig) -> Self { + let client = PodmanClient::new(config.socket_path.clone()); + Self { + client, + config, + network_gateway_ip: None, + } + } +} + +/// Check whether the current user has subuid/subgid ranges configured. +/// +/// Rootless Podman requires entries in `/etc/subuid` and `/etc/subgid` for +/// the running user. If missing, container creation fails with an obscure +/// error. This pre-flight check emits a warning to guide operators. +fn check_subuid_range() { + let uid = nix::unistd::getuid().as_raw(); + let username = nix::unistd::User::from_uid(nix::unistd::Uid::from_raw(uid)) + .ok() + .flatten() + .map(|u| u.name); + + let has_range = |path: &str| -> bool { + let Ok(content) = std::fs::read_to_string(path) else { + return false; + }; + let uid_str = uid.to_string(); + content.lines().any(|line| { + let Some(entry) = line.split(':').next() else { + return false; + }; + entry == uid_str || username.as_deref() == Some(entry) + }) + }; + + if !has_range("/etc/subuid") || !has_range("/etc/subgid") { + let user_display = username.as_deref().map_or_else( + || format!("UID {uid}"), + |name| format!("{name} (UID {uid})"), + ); + warn!( + user = %user_display, + "Rootless Podman detected but no /etc/subuid or /etc/subgid entry found. \ + Container creation may fail. Add entries with: \ + sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $(whoami)" + ); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use http_body_util::Full; + use hyper::body::Bytes; + use hyper::server::conn::http1; + use hyper::service::service_fn; + use hyper::{Response, StatusCode}; + use hyper_util::rt::TokioIo; + use std::collections::VecDeque; + use std::convert::Infallible; + use std::path::PathBuf; + use std::sync::{Arc, Mutex}; + use std::time::{SystemTime, UNIX_EPOCH}; + use tokio::net::UnixListener; + + #[test] + fn podman_driver_error_from_conflict() { + let err = ComputeDriverError::from(PodmanApiError::Conflict("exists".into())); + assert!(matches!(err, ComputeDriverError::AlreadyExists)); + } + + #[test] + fn podman_driver_error_from_not_found() { + let err = ComputeDriverError::from(PodmanApiError::NotFound("gone".into())); + assert!(matches!(err, ComputeDriverError::Message(_))); + } + + // ── grpc_endpoint auto-detection ─────────────────────────────────── + // + // PodmanComputeDriver::new() fills grpc_endpoint when it is empty. + // The scheme (http vs https) depends on whether TLS client certs are + // configured. These tests simulate the auto-detection logic. + + #[test] + fn grpc_endpoint_http_without_tls() { + let mut cfg = PodmanComputeConfig { + gateway_port: 8081, + ..PodmanComputeConfig::default() + }; + if cfg.grpc_endpoint.is_empty() { + let scheme = if cfg.tls_enabled() { "https" } else { "http" }; + cfg.grpc_endpoint = format!("{scheme}://host.containers.internal:{}", cfg.gateway_port); + } + assert_eq!(cfg.grpc_endpoint, "http://host.containers.internal:8081"); + } + + #[test] + fn grpc_endpoint_https_with_tls() { + let mut cfg = PodmanComputeConfig { + gateway_port: 8080, + guest_tls_ca: Some(PathBuf::from("/tls/ca.crt")), + guest_tls_cert: Some(PathBuf::from("/tls/tls.crt")), + guest_tls_key: Some(PathBuf::from("/tls/tls.key")), + ..PodmanComputeConfig::default() + }; + if cfg.grpc_endpoint.is_empty() { + let scheme = if cfg.tls_enabled() { "https" } else { "http" }; + cfg.grpc_endpoint = format!("{scheme}://host.containers.internal:{}", cfg.gateway_port); + } + assert_eq!(cfg.grpc_endpoint, "https://host.containers.internal:8080"); + } + + #[test] + fn partial_tls_config_returns_error() { + let cfg = PodmanComputeConfig { + gateway_port: 8080, + guest_tls_ca: Some(PathBuf::from("/tls/ca.crt")), + // guest_tls_cert and guest_tls_key not set — incomplete TLS config. + ..PodmanComputeConfig::default() + }; + assert!(!cfg.tls_enabled()); + let err = cfg + .validate_tls_config() + .expect_err("partial TLS config should be rejected"); + let msg = err.to_string(); + assert!( + msg.contains("OPENSHELL_PODMAN_TLS_CERT"), + "error should name the missing cert: {msg}" + ); + assert!( + msg.contains("OPENSHELL_PODMAN_TLS_KEY"), + "error should name the missing key: {msg}" + ); + } + + #[test] + fn explicit_grpc_endpoint_takes_precedence() { + let mut cfg = PodmanComputeConfig { + grpc_endpoint: "https://gateway.internal:9000".to_string(), + gateway_port: 8081, + ..PodmanComputeConfig::default() + }; + if cfg.grpc_endpoint.is_empty() { + let scheme = if cfg.tls_enabled() { "https" } else { "http" }; + cfg.grpc_endpoint = format!("{scheme}://host.containers.internal:{}", cfg.gateway_port); + } + assert_eq!(cfg.grpc_endpoint, "https://gateway.internal:9000"); + } + + #[derive(Clone)] + struct StubResponse { + status: StatusCode, + body: String, + } + + impl StubResponse { + fn new(status: StatusCode, body: impl Into) -> Self { + Self { + status, + body: body.into(), + } + } + } + + fn unique_socket_path(test_name: &str) -> PathBuf { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("clock should be after unix epoch") + .as_nanos(); + PathBuf::from(format!( + "/tmp/openshell-podman-{test_name}-{}-{nanos}.sock", + std::process::id() + )) + } + + fn test_driver(socket_path: PathBuf) -> PodmanComputeDriver { + let config = PodmanComputeConfig { + socket_path, + stop_timeout_secs: 10, + ..PodmanComputeConfig::default() + }; + PodmanComputeDriver::for_tests(config) + } + + fn api_path(path: &str) -> String { + format!("/v5.0.0{path}") + } + + fn spawn_podman_stub( + test_name: &str, + responses: Vec, + ) -> ( + PathBuf, + Arc>>, + tokio::task::JoinHandle<()>, + ) { + let socket_path = unique_socket_path(test_name); + let _ = std::fs::remove_file(&socket_path); + let listener = UnixListener::bind(&socket_path).expect("test socket should bind"); + let request_log = Arc::new(Mutex::new(Vec::new())); + let response_queue = Arc::new(Mutex::new(VecDeque::from(responses))); + let expected = response_queue + .lock() + .expect("response queue lock should not be poisoned") + .len(); + let socket_path_for_task = socket_path.clone(); + let log_for_task = request_log.clone(); + let queue_for_task = response_queue; + let handle = tokio::spawn(async move { + for _ in 0..expected { + let (stream, _) = listener.accept().await.expect("test stub should accept"); + let log = log_for_task.clone(); + let queue = queue_for_task.clone(); + let result = http1::Builder::new() + .serve_connection( + TokioIo::new(stream), + service_fn(move |req| { + let log = log.clone(); + let queue = queue.clone(); + async move { + let path = req.uri().path_and_query().map_or_else( + || req.uri().path().to_string(), + |pq| pq.as_str().to_string(), + ); + log.lock() + .expect("request log lock should not be poisoned") + .push(format!("{} {}", req.method(), path)); + let response = queue + .lock() + .expect("response queue lock should not be poisoned") + .pop_front() + .expect("stub response should exist"); + Ok::<_, Infallible>( + Response::builder() + .status(response.status) + .body(Full::new(Bytes::from(response.body))) + .expect("stub response should build"), + ) + } + }), + ) + .await; + // The one-shot test client can close the Unix socket after the + // response, which Hyper reports as a shutdown error. Let the + // request log assertions below decide whether the stub served + // the expected API calls. + let _ = result; + } + let _ = std::fs::remove_file(&socket_path_for_task); + }); + (socket_path, request_log, handle) + } + + #[tokio::test] + async fn delete_sandbox_cleans_up_with_request_id_when_container_is_already_gone() { + let sandbox_id = "sandbox-123"; + let sandbox_name = "demo"; + let container_name = container::container_name(sandbox_name); + let volume_name = container::volume_name(sandbox_id); + let secret_name = container::secret_name(sandbox_id); + let (socket_path, request_log, handle) = spawn_podman_stub( + "delete-not-found", + vec![ + StubResponse::new(StatusCode::NOT_FOUND, r#"{"message":"gone"}"#), + StubResponse::new(StatusCode::NOT_FOUND, r#"{"message":"gone"}"#), + StubResponse::new(StatusCode::NOT_FOUND, r#"{"message":"gone"}"#), + StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new(StatusCode::NO_CONTENT, ""), + ], + ); + let driver = test_driver(socket_path.clone()); + + let deleted = driver + .delete_sandbox(sandbox_id, sandbox_name) + .await + .expect("delete should succeed"); + + assert!(!deleted, "missing container should report deleted=false"); + handle.await.expect("stub task should finish"); + let requests = request_log + .lock() + .expect("request log lock should not be poisoned") + .clone(); + assert_eq!( + requests, + vec![ + format!( + "GET {}", + api_path(&format!("/libpod/containers/{container_name}/json")) + ), + format!( + "POST {}", + api_path(&format!( + "/libpod/containers/{container_name}/stop?timeout=10" + )) + ), + format!( + "DELETE {}", + api_path(&format!( + "/libpod/containers/{container_name}?force=true&v=true" + )) + ), + format!( + "DELETE {}", + api_path(&format!("/libpod/volumes/{volume_name}")) + ), + format!( + "DELETE {}", + api_path(&format!("/libpod/secrets/{secret_name}")) + ), + ] + ); + let _ = std::fs::remove_file(socket_path); + } + + #[tokio::test] + async fn delete_sandbox_uses_request_id_when_container_label_disagrees() { + let sandbox_id = "sandbox-request-id"; + let sandbox_name = "demo"; + let container_name = container::container_name(sandbox_name); + let volume_name = container::volume_name(sandbox_id); + let secret_name = container::secret_name(sandbox_id); + let inspect_body = serde_json::json!({ + "Id": "container-id", + "Name": format!("/{container_name}"), + "State": { + "Status": "running", + "Running": true + }, + "Config": { + "Labels": { + LABEL_SANDBOX_ID: "sandbox-label-id" + } + } + }) + .to_string(); + let (socket_path, request_log, handle) = spawn_podman_stub( + "delete-mismatch", + vec![ + StubResponse::new(StatusCode::OK, inspect_body), + StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new(StatusCode::NO_CONTENT, ""), + ], + ); + let driver = test_driver(socket_path.clone()); + + let deleted = driver + .delete_sandbox(sandbox_id, sandbox_name) + .await + .expect("delete should succeed"); + + assert!(deleted, "existing container should report deleted=true"); + handle.await.expect("stub task should finish"); + let requests = request_log + .lock() + .expect("request log lock should not be poisoned") + .clone(); + assert_eq!( + requests[3..], + [ + format!( + "DELETE {}", + api_path(&format!("/libpod/volumes/{volume_name}")) + ), + format!( + "DELETE {}", + api_path(&format!("/libpod/secrets/{secret_name}")) + ), + ] + ); + let _ = std::fs::remove_file(socket_path); + } +} diff --git a/crates/openshell-driver-podman/src/grpc.rs b/crates/openshell-driver-podman/src/grpc.rs new file mode 100644 index 0000000000..df4c90d13f --- /dev/null +++ b/crates/openshell-driver-podman/src/grpc.rs @@ -0,0 +1,415 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +#![allow(clippy::result_large_err)] // gRPC handlers return Result<_, tonic::Status> + +use futures::{Stream, StreamExt}; +use openshell_core::proto::compute::v1::{ + CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, + GetCapabilitiesRequest, GetCapabilitiesResponse, GetSandboxRequest, GetSandboxResponse, + ListSandboxesRequest, ListSandboxesResponse, StopSandboxRequest, StopSandboxResponse, + ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesEvent, + WatchSandboxesRequest, compute_driver_server::ComputeDriver, +}; +use std::pin::Pin; +use tonic::{Request, Response, Status}; + +use crate::PodmanComputeDriver; +use openshell_core::ComputeDriverError; + +#[derive(Debug, Clone)] +pub struct ComputeDriverService { + driver: PodmanComputeDriver, +} + +impl ComputeDriverService { + #[must_use] + pub fn new(driver: PodmanComputeDriver) -> Self { + Self { driver } + } +} + +#[tonic::async_trait] +impl ComputeDriver for ComputeDriverService { + async fn get_capabilities( + &self, + _request: Request, + ) -> Result, Status> { + self.driver + .capabilities() + .map(Response::new) + .map_err(status_from_driver_error) + } + + async fn validate_sandbox_create( + &self, + request: Request, + ) -> Result, Status> { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + self.driver + .validate_sandbox_create(&sandbox) + .map_err(status_from_driver_error)?; + Ok(Response::new(ValidateSandboxCreateResponse {})) + } + + async fn get_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + if request.sandbox_name.is_empty() { + return Err(Status::invalid_argument("sandbox_name is required")); + } + + let sandbox = self + .driver + .get_sandbox(&request.sandbox_name) + .await + .map_err(status_from_driver_error)? + .ok_or_else(|| Status::not_found("sandbox not found"))?; + + if !request.sandbox_id.is_empty() && request.sandbox_id != sandbox.id { + return Err(Status::failed_precondition( + "sandbox_id did not match the fetched sandbox", + )); + } + + Ok(Response::new(GetSandboxResponse { + sandbox: Some(sandbox), + })) + } + + async fn list_sandboxes( + &self, + _request: Request, + ) -> Result, Status> { + let sandboxes = self + .driver + .list_sandboxes() + .await + .map_err(status_from_driver_error)?; + Ok(Response::new(ListSandboxesResponse { sandboxes })) + } + + async fn create_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + self.driver + .create_sandbox(&sandbox) + .await + .map_err(status_from_driver_error)?; + Ok(Response::new(CreateSandboxResponse {})) + } + + async fn stop_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + if request.sandbox_name.is_empty() { + return Err(Status::invalid_argument("sandbox_name is required")); + } + self.driver + .stop_sandbox(&request.sandbox_name) + .await + .map_err(status_from_driver_error)?; + Ok(Response::new(StopSandboxResponse {})) + } + + async fn delete_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + if request.sandbox_name.is_empty() { + return Err(Status::invalid_argument("sandbox_name is required")); + } + let deleted = self + .driver + .delete_sandbox(&request.sandbox_id, &request.sandbox_name) + .await + .map_err(status_from_driver_error)?; + Ok(Response::new(DeleteSandboxResponse { deleted })) + } + + type WatchSandboxesStream = + Pin> + Send + 'static>>; + + async fn watch_sandboxes( + &self, + _request: Request, + ) -> Result, Status> { + let stream = self + .driver + .watch_sandboxes() + .await + .map_err(status_from_driver_error)?; + let stream = stream.map(|item| item.map_err(|err| Status::internal(err.to_string()))); + Ok(Response::new(Box::pin(stream))) + } +} + +fn status_from_driver_error(err: ComputeDriverError) -> Status { + match err { + ComputeDriverError::AlreadyExists => Status::already_exists("sandbox already exists"), + ComputeDriverError::Precondition(message) => Status::failed_precondition(message), + ComputeDriverError::Message(message) => Status::internal(message), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::config::PodmanComputeConfig; + use crate::container; + use http_body_util::Full; + use hyper::body::Bytes; + use hyper::server::conn::http1; + use hyper::service::service_fn; + use hyper::{Response as HyperResponse, StatusCode}; + use hyper_util::rt::TokioIo; + use std::collections::VecDeque; + use std::convert::Infallible; + use std::path::PathBuf; + use std::sync::{Arc, Mutex}; + use std::time::{SystemTime, UNIX_EPOCH}; + + #[test] + fn precondition_driver_errors_map_to_failed_precondition_status() { + let status = status_from_driver_error(ComputeDriverError::Precondition( + "sandbox container is not running".to_string(), + )); + + assert_eq!(status.code(), tonic::Code::FailedPrecondition); + assert_eq!(status.message(), "sandbox container is not running"); + } + + #[test] + fn already_exists_driver_errors_map_to_already_exists_status() { + let status = status_from_driver_error(ComputeDriverError::AlreadyExists); + assert_eq!(status.code(), tonic::Code::AlreadyExists); + } + + #[derive(Clone)] + struct StubResponse { + status: StatusCode, + body: String, + } + + impl StubResponse { + fn new(status: StatusCode, body: impl Into) -> Self { + Self { + status, + body: body.into(), + } + } + } + + fn unique_socket_path(test_name: &str) -> PathBuf { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("clock should be after unix epoch") + .as_nanos(); + PathBuf::from(format!( + "/tmp/openshell-podman-grpc-{test_name}-{}-{nanos}.sock", + std::process::id() + )) + } + + fn spawn_podman_stub( + test_name: &str, + responses: Vec, + ) -> ( + PathBuf, + Arc>>, + tokio::task::JoinHandle<()>, + ) { + let socket_path = unique_socket_path(test_name); + let _ = std::fs::remove_file(&socket_path); + let listener = + tokio::net::UnixListener::bind(&socket_path).expect("test socket should bind"); + let request_log = Arc::new(Mutex::new(Vec::new())); + let response_queue = Arc::new(Mutex::new(VecDeque::from(responses))); + let expected = response_queue + .lock() + .expect("response queue lock should not be poisoned") + .len(); + let socket_path_for_task = socket_path.clone(); + let log_for_task = request_log.clone(); + let queue_for_task = response_queue; + let handle = tokio::spawn(async move { + for _ in 0..expected { + let (stream, _) = listener.accept().await.expect("test stub should accept"); + let log = log_for_task.clone(); + let queue = queue_for_task.clone(); + let result = http1::Builder::new() + .serve_connection( + TokioIo::new(stream), + service_fn(move |req| { + let log = log.clone(); + let queue = queue.clone(); + async move { + let path = req.uri().path_and_query().map_or_else( + || req.uri().path().to_string(), + |pq| pq.as_str().to_string(), + ); + log.lock() + .expect("request log lock should not be poisoned") + .push(format!("{} {}", req.method(), path)); + let response = queue + .lock() + .expect("response queue lock should not be poisoned") + .pop_front() + .expect("stub response should exist"); + Ok::<_, Infallible>( + HyperResponse::builder() + .status(response.status) + .body(Full::new(Bytes::from(response.body))) + .expect("stub response should build"), + ) + } + }), + ) + .await; + // The one-shot test client can close the Unix socket after the + // response, which Hyper reports as a shutdown error. Let the + // request log assertions below decide whether the stub served + // the expected API calls. + let _ = result; + } + let _ = std::fs::remove_file(&socket_path_for_task); + }); + (socket_path, request_log, handle) + } + + fn test_service(socket_path: PathBuf) -> ComputeDriverService { + let config = PodmanComputeConfig { + socket_path, + stop_timeout_secs: 10, + ..PodmanComputeConfig::default() + }; + ComputeDriverService::new(PodmanComputeDriver::for_tests(config)) + } + + fn api_path(path: &str) -> String { + format!("/v5.0.0{path}") + } + + #[tokio::test] + async fn delete_sandbox_rejects_missing_sandbox_name() { + let service = test_service(unique_socket_path("missing-name")); + + let err = ComputeDriver::delete_sandbox( + &service, + Request::new(DeleteSandboxRequest { + sandbox_id: "sandbox-123".to_string(), + sandbox_name: String::new(), + }), + ) + .await + .expect_err("missing sandbox_name should fail"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert_eq!(err.message(), "sandbox_name is required"); + } + + #[tokio::test] + async fn delete_sandbox_rejects_missing_sandbox_id() { + let service = test_service(unique_socket_path("missing-id")); + + let err = ComputeDriver::delete_sandbox( + &service, + Request::new(DeleteSandboxRequest { + sandbox_id: String::new(), + sandbox_name: "demo".to_string(), + }), + ) + .await + .expect_err("missing sandbox_id should fail"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert_eq!(err.message(), "sandbox_id is required"); + } + + #[tokio::test] + async fn delete_sandbox_forwards_request_sandbox_id_to_driver_cleanup() { + let sandbox_id = "sandbox-abc"; + let sandbox_name = "demo"; + let container_name = container::container_name(sandbox_name); + let volume_name = container::volume_name(sandbox_id); + let secret_name = container::secret_name(sandbox_id); + let (socket_path, request_log, handle) = spawn_podman_stub( + "forward-id", + vec![ + StubResponse::new(StatusCode::NOT_FOUND, r#"{"message":"gone"}"#), + StubResponse::new(StatusCode::NOT_FOUND, r#"{"message":"gone"}"#), + StubResponse::new(StatusCode::NOT_FOUND, r#"{"message":"gone"}"#), + StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new(StatusCode::NO_CONTENT, ""), + ], + ); + let service = test_service(socket_path.clone()); + + let response = ComputeDriver::delete_sandbox( + &service, + Request::new(DeleteSandboxRequest { + sandbox_id: sandbox_id.to_string(), + sandbox_name: sandbox_name.to_string(), + }), + ) + .await + .expect("delete should succeed") + .into_inner(); + + assert!( + !response.deleted, + "already-removed containers should still report deleted=false" + ); + handle.await.expect("stub task should finish"); + let requests = request_log + .lock() + .expect("request log lock should not be poisoned") + .clone(); + assert_eq!( + requests, + vec![ + format!( + "GET {}", + api_path(&format!("/libpod/containers/{container_name}/json")) + ), + format!( + "POST {}", + api_path(&format!( + "/libpod/containers/{container_name}/stop?timeout=10" + )) + ), + format!( + "DELETE {}", + api_path(&format!( + "/libpod/containers/{container_name}?force=true&v=true" + )) + ), + format!( + "DELETE {}", + api_path(&format!("/libpod/volumes/{volume_name}")) + ), + format!( + "DELETE {}", + api_path(&format!("/libpod/secrets/{secret_name}")) + ), + ] + ); + let _ = std::fs::remove_file(socket_path); + } +} diff --git a/crates/openshell-driver-podman/src/lib.rs b/crates/openshell-driver-podman/src/lib.rs new file mode 100644 index 0000000000..630deaee1e --- /dev/null +++ b/crates/openshell-driver-podman/src/lib.rs @@ -0,0 +1,13 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +pub(crate) mod client; +pub mod config; +pub(crate) mod container; +pub mod driver; +pub mod grpc; +pub(crate) mod watcher; + +pub use config::PodmanComputeConfig; +pub use driver::PodmanComputeDriver; +pub use grpc::ComputeDriverService; diff --git a/crates/openshell-driver-podman/src/main.rs b/crates/openshell-driver-podman/src/main.rs new file mode 100644 index 0000000000..9095915ddf --- /dev/null +++ b/crates/openshell-driver-podman/src/main.rs @@ -0,0 +1,145 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use clap::Parser; +use miette::{IntoDiagnostic, Result}; +use std::net::SocketAddr; +use std::path::PathBuf; +use tracing::info; +use tracing_subscriber::EnvFilter; + +use openshell_core::VERSION; +use openshell_core::config::{ + DEFAULT_NETWORK_NAME, DEFAULT_SSH_HANDSHAKE_SKEW_SECS, DEFAULT_SSH_PORT, + DEFAULT_STOP_TIMEOUT_SECS, +}; +use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; +use openshell_driver_podman::config::ImagePullPolicy; +use openshell_driver_podman::{ComputeDriverService, PodmanComputeConfig, PodmanComputeDriver}; + +#[derive(Parser)] +#[command(name = "openshell-driver-podman")] +#[command(version = VERSION)] +struct Args { + #[arg( + long, + env = "OPENSHELL_COMPUTE_DRIVER_BIND", + default_value = "127.0.0.1:50061" + )] + bind_address: SocketAddr, + + #[arg(long, env = "OPENSHELL_LOG_LEVEL", default_value = "info")] + log_level: String, + + /// Path to the Podman API Unix socket. + #[arg(long, env = "OPENSHELL_PODMAN_SOCKET")] + podman_socket: Option, + + #[arg(long, env = "OPENSHELL_SANDBOX_IMAGE")] + sandbox_image: Option, + + #[arg( + long, + env = "OPENSHELL_SANDBOX_IMAGE_PULL_POLICY", + default_value_t = ImagePullPolicy::Missing + )] + sandbox_image_pull_policy: ImagePullPolicy, + + #[arg(long, env = "OPENSHELL_GRPC_ENDPOINT")] + grpc_endpoint: Option, + + /// Port the gateway server is listening on. + /// + /// Used when `--grpc-endpoint` is not set to auto-detect the endpoint + /// that sandbox containers dial back to. + #[arg( + long, + env = "OPENSHELL_GATEWAY_PORT", + default_value_t = openshell_core::config::DEFAULT_SERVER_PORT + )] + gateway_port: u16, + + #[arg( + long, + env = "OPENSHELL_SANDBOX_SSH_SOCKET_PATH", + default_value = "/run/openshell/ssh.sock" + )] + sandbox_ssh_socket_path: String, + + /// Podman bridge network name. + #[arg(long, env = "OPENSHELL_NETWORK_NAME", default_value = DEFAULT_NETWORK_NAME)] + network_name: String, + + #[arg(long, env = "OPENSHELL_SANDBOX_SSH_PORT", default_value_t = DEFAULT_SSH_PORT)] + sandbox_ssh_port: u16, + + #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SECRET")] + ssh_handshake_secret: String, + + #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SKEW_SECS", default_value_t = DEFAULT_SSH_HANDSHAKE_SKEW_SECS)] + ssh_handshake_skew_secs: u64, + + /// Container stop timeout in seconds (SIGTERM → SIGKILL). + #[arg(long, env = "OPENSHELL_STOP_TIMEOUT", default_value_t = DEFAULT_STOP_TIMEOUT_SECS)] + stop_timeout: u32, + + /// OCI image containing the openshell-sandbox supervisor binary. + #[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE")] + supervisor_image: String, + + /// Host path to the CA certificate for sandbox mTLS. + #[arg(long, env = "OPENSHELL_PODMAN_TLS_CA")] + podman_tls_ca: Option, + + /// Host path to the client certificate for sandbox mTLS. + #[arg(long, env = "OPENSHELL_PODMAN_TLS_CERT")] + podman_tls_cert: Option, + + /// Host path to the client private key for sandbox mTLS. + #[arg(long, env = "OPENSHELL_PODMAN_TLS_KEY")] + podman_tls_key: Option, +} + +#[tokio::main] +async fn main() -> Result<()> { + let args = Args::parse(); + tracing_subscriber::fmt() + .with_env_filter( + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)), + ) + .init(); + + let socket_path = args + .podman_socket + .unwrap_or_else(PodmanComputeConfig::default_socket_path); + + let driver = PodmanComputeDriver::new(PodmanComputeConfig { + socket_path, + default_image: args.sandbox_image.unwrap_or_default(), + image_pull_policy: args.sandbox_image_pull_policy, + grpc_endpoint: args.grpc_endpoint.unwrap_or_default(), + gateway_port: args.gateway_port, + sandbox_ssh_socket_path: args.sandbox_ssh_socket_path, + network_name: args.network_name, + ssh_port: args.sandbox_ssh_port, + ssh_handshake_secret: args.ssh_handshake_secret, + ssh_handshake_skew_secs: args.ssh_handshake_skew_secs, + stop_timeout_secs: args.stop_timeout, + supervisor_image: args.supervisor_image, + guest_tls_ca: args.podman_tls_ca, + guest_tls_cert: args.podman_tls_cert, + guest_tls_key: args.podman_tls_key, + }) + .await + .into_diagnostic()?; + + info!(address = %args.bind_address, "Starting Podman compute driver"); + tonic::transport::Server::builder() + .add_service(ComputeDriverServer::new(ComputeDriverService::new(driver))) + .serve_with_shutdown(args.bind_address, async { + tokio::signal::ctrl_c().await.ok(); + info!("Received shutdown signal, draining in-flight requests"); + }) + .await + .into_diagnostic() +} diff --git a/crates/openshell-driver-podman/src/watcher.rs b/crates/openshell-driver-podman/src/watcher.rs new file mode 100644 index 0000000000..54606ea442 --- /dev/null +++ b/crates/openshell-driver-podman/src/watcher.rs @@ -0,0 +1,549 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Maps Podman container events to the compute-driver watch protocol. + +use crate::client::{ + ContainerInspect, ContainerListEntry, ContainerState, HealthState, PodmanApiError, + PodmanClient, PodmanEvent, +}; +use crate::container::{LABEL_MANAGED_FILTER, LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, short_id}; +use futures::Stream; +use openshell_core::ComputeDriverError; +use openshell_core::proto::compute::v1::{ + DriverCondition, DriverSandbox, DriverSandboxStatus, WatchSandboxesDeletedEvent, + WatchSandboxesEvent, WatchSandboxesSandboxEvent, watch_sandboxes_event, +}; +use std::pin::Pin; +use tokio::sync::mpsc; +use tokio_stream::wrappers::ReceiverStream; +use tracing::{debug, info, warn}; + +// Condition reason constants shared across event-building paths. +const CONDITION_RUNNING: &str = "ContainerRunning"; +const CONDITION_STARTING: &str = "ContainerStarting"; +const CONDITION_EXITED: &str = "ContainerExited"; +const CONDITION_STOPPED: &str = "ContainerStopped"; + +pub type WatchStream = + Pin> + Send>>; + +/// Build a `WatchSandboxesEvent` carrying a sandbox snapshot. +fn sandbox_event(sandbox: DriverSandbox) -> WatchSandboxesEvent { + WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { + sandbox: Some(sandbox), + }, + )), + } +} + +/// Build a `WatchSandboxesEvent` for a deleted sandbox. +fn deleted_event(sandbox_id: String) -> WatchSandboxesEvent { + WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Deleted( + WatchSandboxesDeletedEvent { sandbox_id }, + )), + } +} + +/// Start a watch stream that emits current state and live events. +/// +/// The stream first emits a snapshot of all currently-running managed +/// sandboxes (initial state sync), then delivers live container events +/// as they arrive from the Podman event stream. +/// +/// # Reconnection contract +/// +/// The returned stream is **single-use**. When the Podman event connection +/// drops (daemon restart, socket error, or clean shutdown), the stream +/// terminates with a final error item and stops producing events. +/// +/// Callers are responsible for reconnecting by calling [`start_watch`] again. +/// The server's `ComputeRuntime::watch_loop` in `openshell-server` provides +/// this behaviour with a 2-second backoff: when the stream terminates with an +/// error, `watch_loop` sleeps and then calls `watch_sandboxes()` again, which +/// ultimately calls `start_watch()` again and re-syncs state. +/// +/// **Do not add reconnection logic inside this function.** A local reconnect +/// would race with `watch_loop`'s retry and produce duplicate initial-sync +/// events that corrupt the server's sandbox index. +pub async fn start_watch(client: PodmanClient) -> Result { + let (tx, rx) = mpsc::channel::>(256); + + // 1. Subscribe to events first so we don't miss any during the list. + let mut event_rx = client.events_stream(LABEL_MANAGED_FILTER).await?; + + // 2. List existing containers for initial state sync. + let existing = client.list_containers(LABEL_MANAGED_FILTER).await?; + + for entry in &existing { + // For running containers, use inspect to get full state including + // health check status — matching the same condition derivation used + // for live events. + if entry.state == "running" { + match client.inspect_container(&entry.id).await { + Ok(inspect) => { + if let Some(sandbox) = driver_sandbox_from_inspect(&inspect) { + if tx.send(Ok(sandbox_event(sandbox))).await.is_err() { + return Err(PodmanApiError::Connection( + "watch receiver dropped during initial sync".into(), + )); + } + continue; + } + } + Err(e) => { + warn!( + container_id = %entry.id, + error = %e, + "Failed to inspect running container during initial sync, falling back to list entry" + ); + } + } + } + if let Some(event) = driver_sandbox_from_list_entry(entry).map(sandbox_event) + && tx.send(Ok(event)).await.is_err() + { + return Err(PodmanApiError::Connection( + "watch receiver dropped during initial sync".into(), + )); + } + } + + // 3. Stream live events (buffered during the list operation above). + + tokio::spawn(async move { + while let Some(result) = event_rx.recv().await { + match result { + Ok(event) => { + if let Some(we) = map_podman_event(&event, &client).await + && tx.send(Ok(we)).await.is_err() + { + return; + } + } + Err(e) => { + if tx + .send(Err(ComputeDriverError::Message(e.to_string()))) + .await + .is_err() + { + return; + } + } + } + } + // The Podman event stream has ended — either because the Podman + // daemon restarted, the Unix socket was closed, or the connection + // dropped. We do NOT reconnect here; see the doc-comment on + // start_watch() for the full reconnection contract. + // + // Sending this error terminates the WatchStream seen by the caller. + // The server's watch_loop detects the terminal error, waits 2 seconds, + // then calls watch_sandboxes() → start_watch() again, which re-lists + // all containers and re-subscribes to events. This ensures a full + // state resync after any Podman daemon interruption. + warn!("podman event stream ended unexpectedly; watch_loop will reconnect"); + let _ = tx + .send(Err(ComputeDriverError::Message( + "podman event stream ended unexpectedly".to_string(), + ))) + .await; + }); + + Ok(Box::pin(ReceiverStream::new(rx))) +} + +/// Map a Podman event to an optional watch event. +/// +/// Some events (like `remove`) produce a deletion event. State-change events +/// trigger a container inspect to build a full snapshot. +async fn map_podman_event( + event: &PodmanEvent, + client: &PodmanClient, +) -> Option { + let container_id = &event.actor.id; + let sandbox_id = event + .actor + .attributes + .get(LABEL_SANDBOX_ID) + .cloned() + .unwrap_or_default(); + + if sandbox_id.is_empty() { + debug!( + container_id = %container_id, + action = %event.action, + "Ignoring event for container without sandbox-id label" + ); + return None; + } + + match event.action.as_str() { + "remove" => Some(deleted_event(sandbox_id.clone())), + "create" | "start" | "stop" | "die" | "health_status" => { + // Inspect the container to get current state. + match client.inspect_container(container_id).await { + Ok(inspect) => driver_sandbox_from_inspect(&inspect).map(sandbox_event), + Err(PodmanApiError::NotFound(_)) => { + // The container is already gone by the time we inspected + // it. This is a normal race between the `die`/`stop` event + // and the subsequent `remove` event: Podman fires `die` + // first, but the container may be fully removed before we + // can inspect it. Treat this as a deletion so the server + // does not see a spurious phase regression. + info!( + container_id = %container_id, + action = %event.action, + "Container already removed when inspecting after event, emitting deleted event" + ); + Some(deleted_event(sandbox_id.clone())) + } + Err(e) => { + warn!( + container_id = %container_id, + error = %e, + "Failed to inspect container after event" + ); + // Emit a synthetic event with the info we have so the + // server knows something happened. + let sandbox_name = event + .actor + .attributes + .get(LABEL_SANDBOX_NAME) + .cloned() + .unwrap_or_default(); + Some(sandbox_event(build_driver_sandbox( + sandbox_id.clone(), + sandbox_name, + String::new(), + short_id(container_id), + DriverCondition { + r#type: "Ready".to_string(), + status: "Unknown".to_string(), + reason: "InspectFailed".to_string(), + message: format!("Container inspect failed: {e}"), + last_transition_time: String::new(), + }, + false, + ))) + } + } + } + _ => { + debug!(action = %event.action, "Ignoring unhandled Podman event"); + None + } + } +} + +/// Construct a `DriverSandbox` from common fields. +/// +/// Centralises the boilerplate that every event/inspect/list path shares: +/// `namespace`, `spec`, `agent_fd`, and `sandbox_fd` are always empty in +/// the Podman driver. +fn build_driver_sandbox( + sandbox_id: String, + sandbox_name: String, + instance_name: String, + instance_id: String, + condition: DriverCondition, + deleting: bool, +) -> DriverSandbox { + DriverSandbox { + id: sandbox_id, + name: sandbox_name, + namespace: String::new(), + spec: None, + status: Some(DriverSandboxStatus { + sandbox_name: instance_name, + instance_id, + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: vec![condition], + deleting, + }), + } +} + +/// Build a `DriverSandbox` from a container inspection result. +pub fn driver_sandbox_from_inspect(inspect: &ContainerInspect) -> Option { + let sandbox_id = inspect.config.labels.get(LABEL_SANDBOX_ID)?.clone(); + let sandbox_name = inspect + .config + .labels + .get(LABEL_SANDBOX_NAME) + .cloned() + .unwrap_or_default(); + + let condition = condition_from_state(&inspect.state); + let deleting = inspect.state.status == "removing"; + + Some(build_driver_sandbox( + sandbox_id, + sandbox_name, + inspect.name.trim_start_matches('/').to_string(), + short_id(&inspect.id), + condition, + deleting, + )) +} + +/// Build a `DriverSandbox` from a container list entry (no inspect needed). +pub fn driver_sandbox_from_list_entry(entry: &ContainerListEntry) -> Option { + let sandbox_id = entry.labels.get(LABEL_SANDBOX_ID)?.clone(); + let sandbox_name = entry + .labels + .get(LABEL_SANDBOX_NAME) + .cloned() + .unwrap_or_default(); + + let (reason, status_str, message) = match entry.state.as_str() { + "running" => ( + CONDITION_RUNNING, + "True", + "Container is running".to_string(), + ), + "created" => ("ContainerCreated", "False", String::new()), + "exited" => (CONDITION_EXITED, "False", String::new()), + "stopped" => (CONDITION_STOPPED, "False", String::new()), + "removing" => ("ContainerRemoving", "False", String::new()), + _ => ("Unknown", "Unknown", String::new()), + }; + + Some(build_driver_sandbox( + sandbox_id, + sandbox_name, + entry.names.first().cloned().unwrap_or_default(), + short_id(&entry.id), + DriverCondition { + r#type: "Ready".to_string(), + status: status_str.to_string(), + reason: reason.to_string(), + message, + last_transition_time: String::new(), + }, + entry.state == "removing", + )) +} + +/// Derive a `DriverCondition` from Podman container state. +fn condition_from_state(state: &ContainerState) -> DriverCondition { + let (status_val, reason, message) = match state.status.as_str() { + "running" => match &state.health { + Some(HealthState { status }) if status == "healthy" => { + ("True", "HealthCheckPassed", String::new()) + } + Some(HealthState { status }) if status == "unhealthy" => { + ("False", "HealthCheckFailed", String::new()) + } + Some(HealthState { status }) if status == "starting" => { + ("False", "HealthCheckStarting", String::new()) + } + _ => ("False", CONDITION_STARTING, String::new()), + }, + "created" => ("False", "ContainerCreated", String::new()), + "exited" | "stopped" => { + let msg = if state.oom_killed { + "Container was killed by the OOM killer".to_string() + } else { + format!("Container exited with code {}", state.exit_code) + }; + let reason = if state.oom_killed { + "OOMKilled" + } else { + CONDITION_EXITED + }; + ("False", reason, msg) + } + other => ( + "Unknown", + "Unknown", + format!("Unknown container state: {other}"), + ), + }; + + // Use Podman's state timestamps for last_transition_time: + // - Running/healthy states use started_at + // - Stopped/exited states use finished_at + let last_transition_time = match state.status.as_str() { + "running" => state.started_at.clone().unwrap_or_default(), + "exited" | "stopped" => state.finished_at.clone().unwrap_or_default(), + _ => String::new(), + }; + + DriverCondition { + r#type: "Ready".to_string(), + status: status_val.to_string(), + reason: reason.to_string(), + message, + last_transition_time, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn condition_healthy_container() { + let state = ContainerState { + status: "running".to_string(), + running: true, + exit_code: 0, + oom_killed: false, + health: Some(HealthState { + status: "healthy".to_string(), + }), + started_at: Some("2026-04-14T10:00:00Z".to_string()), + finished_at: None, + }; + let cond = condition_from_state(&state); + assert_eq!(cond.r#type, "Ready"); + assert_eq!(cond.status, "True"); + assert_eq!(cond.reason, "HealthCheckPassed"); + assert_eq!(cond.last_transition_time, "2026-04-14T10:00:00Z"); + } + + #[test] + fn condition_oom_killed() { + let state = ContainerState { + status: "exited".to_string(), + running: false, + exit_code: 137, + oom_killed: true, + health: None, + started_at: None, + finished_at: Some("2026-04-14T11:00:00Z".to_string()), + }; + let cond = condition_from_state(&state); + assert_eq!(cond.status, "False"); + assert_eq!(cond.reason, "OOMKilled"); + assert_eq!(cond.last_transition_time, "2026-04-14T11:00:00Z"); + } + + #[test] + fn condition_normal_exit() { + let state = ContainerState { + status: "exited".to_string(), + running: false, + exit_code: 1, + oom_killed: false, + health: None, + started_at: None, + finished_at: Some("2026-04-14T12:00:00Z".to_string()), + }; + let cond = condition_from_state(&state); + assert_eq!(cond.status, "False"); + assert_eq!(cond.reason, "ContainerExited"); + assert!(cond.message.contains("code 1")); + } + + #[test] + fn short_id_truncates() { + assert_eq!(short_id("abc123def456789"), "abc123def456"); + assert_eq!(short_id("short"), "short"); + } + + #[test] + fn sandbox_event_from_list_entry_running() { + let mut labels = std::collections::HashMap::new(); + labels.insert(LABEL_SANDBOX_ID.to_string(), "test-id".to_string()); + labels.insert(LABEL_SANDBOX_NAME.to_string(), "test-name".to_string()); + + let entry = ContainerListEntry { + id: "abc123def456789".to_string(), + names: vec!["openshell-sandbox-test-name".to_string()], + state: "running".to_string(), + labels, + ports: None, + networks: None, + exit_code: 0, + }; + + let sandbox = driver_sandbox_from_list_entry(&entry).expect("should produce a sandbox"); + let status = sandbox.status.expect("should have status"); + assert_eq!(status.conditions.len(), 1); + let cond = &status.conditions[0]; + assert_eq!(cond.status, "True"); + assert_eq!(cond.reason, "ContainerRunning"); + assert!(!status.deleting); + } + + #[test] + fn synthetic_inspect_failed_event_structure() { + // Verify the structure of an inspect-failure event by constructing one + // using the same pattern as the production code. + let condition = DriverCondition { + r#type: "Ready".to_string(), + status: "Unknown".to_string(), + reason: "InspectFailed".to_string(), + message: "Container inspect failed: connection refused".to_string(), + last_transition_time: String::new(), + }; + + let sandbox = DriverSandbox { + id: "sandbox-123".to_string(), + name: "test-sandbox".to_string(), + namespace: String::new(), + spec: None, + status: Some(DriverSandboxStatus { + sandbox_name: String::new(), + instance_id: short_id("container-id-full"), + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: vec![condition], + deleting: false, + }), + }; + + let event = WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { + sandbox: Some(sandbox), + }, + )), + }; + + let payload = event.payload.unwrap(); + let watch_sandboxes_event::Payload::Sandbox(sandbox_event) = payload else { + panic!("expected Sandbox payload") + }; + let status = sandbox_event.sandbox.unwrap().status.unwrap(); + assert_eq!(status.conditions.len(), 1); + let cond = &status.conditions[0]; + assert_eq!(cond.reason, "InspectFailed"); + assert_eq!(cond.status, "Unknown"); + assert!(cond.message.contains("inspect failed")); + } + + #[test] + fn not_found_inspect_produces_deleted_event() { + // When inspect_container returns NotFound (404) after a `die` or + // `stop` event, the container is already gone. The watcher should emit + // a deleted_event rather than a synthetic InspectFailed sandbox event, + // preventing the server from regressing the sandbox phase back to + // Provisioning. + // + // We verify the shape of a deleted_event directly since + // map_podman_event is async and requires a live Podman client. The + // production code path is: + // Err(PodmanApiError::NotFound(_)) => Some(deleted_event(sandbox_id)) + let sandbox_id = "sandbox-abc-123".to_string(); + let event = deleted_event(sandbox_id.clone()); + + let payload = event.payload.expect("deleted event must have a payload"); + match payload { + watch_sandboxes_event::Payload::Deleted(d) => { + assert_eq!(d.sandbox_id, sandbox_id); + } + other => { + panic!( + "expected Deleted payload, got {other:?} — a NotFound inspect must not produce a sandbox or platform event" + ); + } + } + } +} diff --git a/crates/openshell-driver-vm/Cargo.toml b/crates/openshell-driver-vm/Cargo.toml new file mode 100644 index 0000000000..c13d904a61 --- /dev/null +++ b/crates/openshell-driver-vm/Cargo.toml @@ -0,0 +1,56 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-driver-vm" +description = "MicroVM compute driver for OpenShell" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[lib] +name = "openshell_driver_vm" +path = "src/lib.rs" + +[[bin]] +name = "openshell-driver-vm" +path = "src/main.rs" + +[dependencies] +openshell-core = { path = "../openshell-core" } +openshell-vfio = { path = "../openshell-vfio" } + +bollard = { version = "0.20", features = ["ssh"] } +tokio = { workspace = true } +tonic = { workspace = true, features = ["transport"] } +prost-types = { workspace = true } +futures = { workspace = true } +tokio-stream = { workspace = true, features = ["net"] } +nix = { workspace = true } +clap = { workspace = true } +tracing = { workspace = true } +tracing-subscriber = { workspace = true } +miette = { workspace = true } +url = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +oci-client = "0.16" +libc = "0.2" +libloading = "0.8" +tar = "0.4" +flate2 = "1" +sha2 = "0.10" +zstd = "0.13" + +# smol-rs/polling drives the BSD/macOS parent-death detection in +# procguard via kqueue's EVFILT_PROC / NOTE_EXIT filter. We could use +# it on Linux too (via epoll + pidfd) but sticking with +# nix::sys::prctl::set_pdeathsig there keeps the Linux path a single +# syscall with no helper thread. +[target.'cfg(any(target_os = "macos", target_os = "ios", target_os = "freebsd", target_os = "netbsd", target_os = "openbsd", target_os = "dragonfly"))'.dependencies] +polling = "3.11" + +[lints] +workspace = true diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md new file mode 100644 index 0000000000..e71f926edd --- /dev/null +++ b/crates/openshell-driver-vm/README.md @@ -0,0 +1,192 @@ +# openshell-driver-vm + +> Status: Experimental. The VM compute driver is under active development and the interface still has VM-specific plumbing that will be generalized. + +Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess, talks to it over a Unix domain socket with the `openshell.compute.v1.ComputeDriver` gRPC surface, and lets it manage per-sandbox microVMs. The runtime (libkrun + libkrunfw + gvproxy) and the sandbox supervisor are embedded directly in the binary; each sandbox guest rootfs is derived from a configured container image at create time. + +## How it fits together + +```mermaid +flowchart LR + subgraph host["Host process"] + gateway["openshell-server
(compute::vm::spawn)"] + driver["openshell-driver-vm
├── libkrun (VM)
├── gvproxy (net)
└── openshell-sandbox.zst"] + gateway <-->|"gRPC over UDS
compute-driver.sock"| driver + end + + subgraph guest["Per-sandbox microVM"] + init["/srv/openshell-vm-
sandbox-init.sh"] + supervisor["/opt/openshell/bin/
openshell-sandbox
(PID 1)"] + init --> supervisor + end + + driver -->|"CreateSandbox
boots via libkrun"| guest + supervisor -.->|"gRPC callback
--grpc-endpoint"| gateway + + client["openshell-cli"] -->|"SSH proxy
127.0.0.1:<port>"| supervisor + client -->|"CreateSandbox / Watch"| gateway +``` + +Sandbox guests execute `/opt/openshell/bin/openshell-sandbox` as PID 1 inside the VM. gvproxy exposes a single inbound SSH port (`host:` → `guest:2222`) and provides virtio-net egress. + +## Quick start (recommended) + +```shell +mise run gateway:vm +``` + +First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/gvproxy and `mise run vm:supervisor` builds the bundled guest supervisor. Subsequent runs are cached. + +By default `mise run gateway:vm`: + +- Listens on plaintext HTTP at `127.0.0.1:18081`. +- Registers the CLI gateway `vm-dev` by writing `~/.config/openshell/gateways/vm-dev/metadata.json`. It does not modify the workspace `.env`. +- Persists the gateway SQLite DB under `.cache/gateway-vm/gateway.db`. +- Places the VM driver state (per-sandbox rootfs + `compute-driver.sock`) under `/tmp/openshell-vm-driver-$USER-vm-dev/` so the AF_UNIX socket path stays under macOS `SUN_LEN`. +- Passes `--driver-dir $PWD/target/debug` so the freshly built `openshell-driver-vm` is used instead of an older installed copy from `~/.local/libexec/openshell`, `/usr/libexec/openshell`, or `/usr/local/libexec`. + +For GPU passthrough (VFIO), pass `-- --gpu` and run with root privileges: + +```shell +sudo -E env "PATH=$PATH" mise run gateway:vm -- --gpu +``` + +See [`architecture/vm-gpu-sandbox-guide.md`](../../architecture/vm-gpu-sandbox-guide.md) for full GPU prerequisites and usage. + +Point the CLI at the gateway with one of: + +```shell +openshell --gateway vm-dev status +openshell gateway select vm-dev # then plain `openshell ` +``` + +Override defaults via environment: + +```shell +# custom port (fails fast if in use) +OPENSHELL_SERVER_PORT=18091 mise run gateway:vm + +# custom CLI gateway name + namespace +OPENSHELL_VM_GATEWAY_NAME=vm-feature-a \ +OPENSHELL_SANDBOX_NAMESPACE=vm-feature-a \ +mise run gateway:vm + +# custom sandbox image +OPENSHELL_SANDBOX_IMAGE=ghcr.io/example/sandbox:latest mise run gateway:vm +``` + +Teardown: + +```shell +rm -rf /tmp/openshell-vm-driver-$USER-vm-dev .cache/gateway-vm +rm -rf "${XDG_CONFIG_HOME:-$HOME/.config}/openshell/gateways/vm-dev" +``` + +## Manual equivalent + +If you want to drive the launch yourself instead of using `mise run gateway:vm` (i.e. `tasks/scripts/gateway-vm.sh`): + +```shell +# 1. Stage runtime artifacts + supervisor bundle into target/vm-runtime-compressed/ +mise run vm:setup +mise run vm:supervisor # if openshell-sandbox.zst is not already present + +# 2. Build both binaries with the staged artifacts embedded +OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \ + cargo build -p openshell-server -p openshell-driver-vm + +# 3. macOS only: codesign the driver for Hypervisor.framework +codesign \ + --entitlements crates/openshell-driver-vm/entitlements.plist \ + --force -s - target/debug/openshell-driver-vm + +# 4. Start the gateway with the VM driver +mkdir -p /tmp/openshell-vm-driver-$USER-vm-dev .cache/gateway-vm +target/debug/openshell-gateway \ + --drivers vm \ + --disable-tls \ + --db-url "sqlite:.cache/gateway-vm/gateway.db?mode=rwc" \ + --driver-dir $PWD/target/debug \ + --sandbox-namespace vm-dev \ + --sandbox-image \ + --grpc-endpoint http://host.containers.internal:18081 \ + --port 18081 \ + --vm-driver-state-dir /tmp/openshell-vm-driver-$USER-vm-dev +``` + +The gateway resolves `openshell-driver-vm` in this order: `--driver-dir`, conventional install locations (`~/.local/libexec/openshell`, `/usr/libexec/openshell`, `/usr/local/libexec/openshell`, `/usr/local/libexec`), then a sibling of the gateway binary. + +## Flags + +| Flag | Env var | Default | Purpose | +|---|---|---|---| +| `--drivers vm` | `OPENSHELL_DRIVERS` | `kubernetes` | Select the VM compute driver. | +| `--grpc-endpoint URL` | `OPENSHELL_GRPC_ENDPOINT` | — | Required. URL the sandbox guest dials to reach the gateway. Use `http://host.containers.internal:` (or `host.docker.internal` / `host.openshell.internal`) so traffic flows through gvproxy's host-loopback NAT (HostIP `192.168.127.254` → host `127.0.0.1`). Loopback URLs like `http://127.0.0.1:` are rewritten automatically by the driver. The bare gateway IP (`192.168.127.1`) only carries gvproxy's own services and will not reach host-bound ports. | +| `--vm-driver-state-dir DIR` | `OPENSHELL_VM_DRIVER_STATE_DIR` | `target/openshell-vm-driver` | Per-sandbox rootfs, console logs, and the `compute-driver.sock` UDS. | +| `--driver-dir DIR` | `OPENSHELL_DRIVER_DIR` | unset | Override the directory searched for `openshell-driver-vm`. | +| `--vm-driver-vcpus N` | `OPENSHELL_VM_DRIVER_VCPUS` | `2` | vCPUs per sandbox. | +| `--vm-driver-mem-mib N` | `OPENSHELL_VM_DRIVER_MEM_MIB` | `2048` | Memory per sandbox, in MiB. | +| `--vm-krun-log-level N` | `OPENSHELL_VM_KRUN_LOG_LEVEL` | `1` | libkrun verbosity (0–5). | +| `--vm-tls-ca PATH` | `OPENSHELL_VM_TLS_CA` | — | CA cert for the guest's mTLS client bundle. Required when `--grpc-endpoint` uses `https://`. | +| `--vm-tls-cert PATH` | `OPENSHELL_VM_TLS_CERT` | — | Guest client certificate. | +| `--vm-tls-key PATH` | `OPENSHELL_VM_TLS_KEY` | — | Guest client private key. | + +See [`openshell-gateway --help`](../openshell-server/src/cli.rs) for the full flag surface shared with the Kubernetes driver. + +## Verifying the gateway + +The gateway is auto-registered by `mise run gateway:vm`. In another terminal: + +```shell +./scripts/bin/openshell status +./scripts/bin/openshell sandbox create --name demo --from +./scripts/bin/openshell sandbox connect demo +``` + +First sandbox takes 10–30 seconds to boot (image fetch/prepare/cache + libkrun + guest init). If `--from` is omitted, the VM driver uses the gateway's configured default sandbox image. Without either `--from` or `--sandbox-image`, VM sandbox creation fails. Subsequent creates reuse the prepared sandbox rootfs. + +## Logs and debugging + +Raise log verbosity for both processes: + +```shell +RUST_LOG=openshell_server=debug,openshell_driver_vm=debug \ + mise run gateway:vm +``` + +The VM guest's serial console is appended to `//console.log`. The `compute-driver.sock` lives at `/compute-driver.sock`; the gateway removes it on clean shutdown via `ManagedDriverProcess::drop`. + +## Prerequisites + +- macOS on Apple Silicon, or Linux on aarch64/x86_64 with KVM +- Rust toolchain +- Guest-supervisor cross-compile toolchain (needed on macOS, and on Linux when host arch ≠ guest arch): + - Matching rustup target: `rustup target add aarch64-unknown-linux-gnu` (or `x86_64-unknown-linux-gnu` for an amd64 guest) + - `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` uses `cargo zigbuild` to cross-compile the in-VM `openshell-sandbox` supervisor binary. +- [mise](https://mise.jdx.dev/) task runner +- Docker-compatible socket on the local CLI/gateway host when using + `openshell sandbox create --from ./Dockerfile` or `--from ./dir`; the CLI + builds the image and the VM driver exports it via the local Docker daemon +- `gh` CLI (used by `mise run vm:setup` to download pre-built runtime artifacts) + +## Releases + +`openshell-driver-vm` is published as a normal OpenShell release artifact: + +- development builds: the rolling `dev` release +- tagged builds: the corresponding `v*` release +- runtime tarballs: the rolling `vm-runtime` release, rebuilt on demand by + `release-vm-kernel.yml` + +On Linux amd64 and arm64, `install-dev.sh` installs the Debian package from the +selected `OPENSHELL_VERSION` release tag. That package includes +`openshell-gateway` and `openshell-driver-vm`. + +## Relationship to `openshell-vm` + +`openshell-vm` is a separate, legacy crate that runs the **whole OpenShell gateway inside a single VM**. It remains in the repository for later deprecation or removal, but is excluded from normal workspace builds and release paths. `openshell-driver-vm` is the active compute driver called by a host-resident gateway to spawn **per-sandbox VMs**. The driver vendors its own rootfs handling and runtime loader so `openshell-server` never has to link libkrun. + +## TODOs + +- The gateway still configures the driver via CLI args; this will move to a gRPC bootstrap call so the driver interface is uniform across backends. See the `TODO(driver-abstraction)` notes in `crates/openshell-server/src/lib.rs` and `crates/openshell-server/src/compute/vm.rs`. +- macOS local builds are codesigned by `tasks/scripts/gateway-vm.sh`; release tarball users must ad-hoc sign `openshell-driver-vm` before running VM sandboxes. diff --git a/crates/openshell-driver-vm/build.rs b/crates/openshell-driver-vm/build.rs new file mode 100644 index 0000000000..ea4c4d2e0d --- /dev/null +++ b/crates/openshell-driver-vm/build.rs @@ -0,0 +1,145 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Build script for openshell-driver-vm. +//! +//! This crate embeds the sandbox supervisor plus the minimal libkrun runtime +//! artifacts it needs to boot VMs without depending on the openshell-vm binary +//! or crate. + +use std::path::{Path, PathBuf}; +use std::{env, fs}; + +fn main() { + println!("cargo:rerun-if-env-changed=OPENSHELL_VM_RUNTIME_COMPRESSED_DIR"); + + if let Ok(dir) = env::var("OPENSHELL_VM_RUNTIME_COMPRESSED_DIR") { + println!("cargo:rerun-if-changed={dir}"); + for name in &[ + "libkrun.so.zst", + "libkrunfw.so.5.zst", + "libkrun.dylib.zst", + "libkrunfw.5.dylib.zst", + "gvproxy.zst", + "openshell-sandbox.zst", + ] { + println!("cargo:rerun-if-changed={dir}/{name}"); + } + } + + let out_dir = PathBuf::from(env::var("OUT_DIR").expect("OUT_DIR not set")); + let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default(); + let target_arch = env::var("CARGO_CFG_TARGET_ARCH").unwrap_or_default(); + + let (libkrun_name, libkrunfw_name) = match target_os.as_str() { + "macos" => ("libkrun.dylib", "libkrunfw.5.dylib"), + "linux" => ("libkrun.so", "libkrunfw.so.5"), + _ => { + println!("cargo:warning=VM runtime not available for {target_os}-{target_arch}"); + generate_stub_resources(&out_dir, &["libkrun", "libkrunfw", "openshell-sandbox.zst"]); + return; + } + }; + + let compressed_dir = if let Ok(dir) = env::var("OPENSHELL_VM_RUNTIME_COMPRESSED_DIR") { + PathBuf::from(dir) + } else { + println!("cargo:warning=OPENSHELL_VM_RUNTIME_COMPRESSED_DIR not set"); + println!("cargo:warning=Run: mise run vm:setup && mise run vm:supervisor"); + generate_stub_resources( + &out_dir, + &[ + &format!("{libkrun_name}.zst"), + &format!("{libkrunfw_name}.zst"), + "gvproxy.zst", + "openshell-sandbox.zst", + ], + ); + return; + }; + + if !compressed_dir.is_dir() { + println!( + "cargo:warning=Compressed runtime dir not found: {}", + compressed_dir.display() + ); + println!("cargo:warning=Run: mise run vm:setup && mise run vm:supervisor"); + generate_stub_resources( + &out_dir, + &[ + &format!("{libkrun_name}.zst"), + &format!("{libkrunfw_name}.zst"), + "gvproxy.zst", + "openshell-sandbox.zst", + ], + ); + return; + } + + let files = [ + (format!("{libkrun_name}.zst"), format!("{libkrun_name}.zst")), + ( + format!("{libkrunfw_name}.zst"), + format!("{libkrunfw_name}.zst"), + ), + ("gvproxy.zst".to_string(), "gvproxy.zst".to_string()), + ( + "openshell-sandbox.zst".to_string(), + "openshell-sandbox.zst".to_string(), + ), + ]; + + let mut all_found = true; + for (src_name, dst_name) in &files { + let src_path = compressed_dir.join(src_name); + let dst_path = out_dir.join(dst_name); + + if !src_path.exists() { + println!( + "cargo:warning=Missing compressed artifact: {}", + src_path.display() + ); + all_found = false; + continue; + } + + if dst_path.exists() { + let _ = fs::remove_file(&dst_path); + } + fs::copy(&src_path, &dst_path).unwrap_or_else(|e| { + panic!( + "Failed to copy {} to {}: {}", + src_path.display(), + dst_path.display(), + e + ) + }); + let size = fs::metadata(&dst_path).map_or(0, |m| m.len()); + println!("cargo:warning=Embedded {src_name}: {size} bytes"); + } + + if !all_found { + println!( + "cargo:warning=Some artifacts missing. Run: mise run vm:setup && mise run vm:supervisor" + ); + generate_stub_resources( + &out_dir, + &[ + &format!("{libkrun_name}.zst"), + &format!("{libkrunfw_name}.zst"), + "gvproxy.zst", + "openshell-sandbox.zst", + ], + ); + } +} + +fn generate_stub_resources(out_dir: &Path, names: &[&str]) { + for name in names { + let path = out_dir.join(name); + if !path.exists() { + fs::write(&path, b"") + .unwrap_or_else(|e| panic!("Failed to write stub {}: {}", path.display(), e)); + } + } +} diff --git a/crates/openshell-driver-vm/entitlements.plist b/crates/openshell-driver-vm/entitlements.plist new file mode 100644 index 0000000000..154f3308ef --- /dev/null +++ b/crates/openshell-driver-vm/entitlements.plist @@ -0,0 +1,8 @@ + + + + + com.apple.security.hypervisor + + + diff --git a/crates/openshell-driver-vm/runtime/README.md b/crates/openshell-driver-vm/runtime/README.md new file mode 100644 index 0000000000..74afeb2be1 --- /dev/null +++ b/crates/openshell-driver-vm/runtime/README.md @@ -0,0 +1,75 @@ +# VM Driver Runtime + +> Status: Experimental. VM support is under active development and may change. + +This directory owns the pinned runtime inputs for `openshell-driver-vm`: + +```text +runtime/ + pins.env + kernel/ + openshell.kconfig +``` + +`openshell-driver-vm` embeds libkrun, libkrunfw, gvproxy, and the bundled +`openshell-sandbox` supervisor. The legacy `crates/openshell-vm` crate remains +in the repository, but normal workspace builds and release workflows do not use +it. + +## Why + +The stock `libkrunfw` kernel does not include the bridge, netfilter, +conntrack, cgroup, seccomp, and Landlock features the sandbox supervisor needs +inside each microVM. `kernel/openshell.kconfig` extends the libkrunfw kernel so +VM sandboxes can run the same supervisor enforcement path as other backends. + +## Build Scripts + +| Script | Platform | Purpose | +|---|---|---| +| `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw and libkrun from source with the custom kernel config | +| `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Builds portable libkrunfw and libkrun from a prebuilt `kernel.c` | +| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages `vm-runtime-.tar.zst` with libraries, gvproxy, and provenance | +| `tasks/scripts/vm/download-kernel-runtime.sh` | Any | Downloads runtime tarballs from the `vm-runtime` release and stages compressed files | + +## Local Flow + +```shell +# Download the current pre-built runtime and stage compressed artifacts +mise run vm:setup + +# Build the bundled guest supervisor +mise run vm:supervisor + +# Build the gateway and VM driver with embedded runtime artifacts +OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \ + cargo build -p openshell-server -p openshell-driver-vm +``` + +Use `FROM_SOURCE=1 mise run vm:setup` to build the runtime from source instead +of downloading `vm-runtime-.tar.zst`. + +## CI Ownership + +`release-vm-kernel.yml` is the on-demand producer for: + +- `vm-runtime-linux-aarch64.tar.zst` +- `vm-runtime-linux-x86_64.tar.zst` +- `vm-runtime-darwin-aarch64.tar.zst` + +Those artifacts stay on the rolling `vm-runtime` release. Normal `dev` and `v*` +release workflows download them, embed them into `openshell-driver-vm`, and +publish the driver binary next to `openshell-gateway`. + +## Provenance + +`package-vm-runtime.sh` writes `provenance.json` into each runtime tarball with +the platform, libkrunfw commit, kernel version, GitHub SHA, and build time. The +driver logs this metadata when it extracts and loads a runtime bundle. + +The release workflow also publishes GitHub artifact attestations for each +runtime tarball. Verify a downloaded runtime with: + +```bash +gh attestation verify vm-runtime-linux-x86_64.tar.zst -R NVIDIA/OpenShell +``` diff --git a/crates/openshell-vm/runtime/kernel/openshell.kconfig b/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig similarity index 100% rename from crates/openshell-vm/runtime/kernel/openshell.kconfig rename to crates/openshell-driver-vm/runtime/kernel/openshell.kconfig diff --git a/crates/openshell-vm/pins.env b/crates/openshell-driver-vm/runtime/pins.env similarity index 97% rename from crates/openshell-vm/pins.env rename to crates/openshell-driver-vm/runtime/pins.env index b3d8022929..4a60c0225f 100644 --- a/crates/openshell-vm/pins.env +++ b/crates/openshell-driver-vm/runtime/pins.env @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Pinned dependency versions for openshell-vm builds. +# Pinned dependency versions for the openshell-driver-vm runtime. # # This file is sourced by build-rootfs.sh and # build-libkrun.sh. It centralises version pins and content-addressed diff --git a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh new file mode 100644 index 0000000000..b61fd49000 --- /dev/null +++ b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh @@ -0,0 +1,419 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Minimal init for sandbox VMs. Runs as PID 1 inside the guest, mounts the +# essential filesystems, configures networking (gvproxy DHCP or TAP static), +# optionally loads NVIDIA GPU drivers, then execs the OpenShell sandbox +# supervisor. + +set -euo pipefail + +# Source QEMU-injected environment variables if present. +if [ -f /srv/openshell-env.sh ]; then + # shellcheck source=/dev/null + source /srv/openshell-env.sh +fi + +BOOT_START=$(date +%s%3N 2>/dev/null || date +%s) +# gvisor-tap-vsock subnet layout: +# 192.168.127.1 — gateway: gvproxy's DNS / DHCP / HTTP API. Does NOT +# proxy arbitrary host ports. +# 192.168.127.254 — host-loopback: NAT-rewritten to host's 127.0.0.1 by +# gvproxy's TCP/UDP/ICMP forwarder. Use this address +# (or any of the host.* hostnames below) to reach a +# service the host is listening on. +# The host.containers.internal / host.docker.internal DNS records served +# by gvproxy's embedded resolver point at 192.168.127.254. We mirror that +# in /etc/hosts so the supervisor can reach the gateway even when +# gvproxy's DNS is not in resolv.conf (e.g. DHCP failed and we fell +# back to 8.8.8.8). +GVPROXY_GATEWAY_IP="192.168.127.1" +GVPROXY_HOST_LOOPBACK_IP="192.168.127.254" +GATEWAY_IP="$GVPROXY_GATEWAY_IP" + +GPU_ENABLED="${GPU_ENABLED:-false}" +VM_NET_IP="${VM_NET_IP:-}" +VM_NET_GW="${VM_NET_GW:-}" +VM_NET_DNS="${VM_NET_DNS:-}" + +ts() { + local now + now=$(date +%s%3N 2>/dev/null || date +%s) + local elapsed=$((now - BOOT_START)) + printf "[%d.%03ds] %s\n" $((elapsed / 1000)) $((elapsed % 1000)) "$*" +} + +parse_endpoint() { + local endpoint="$1" + local scheme rest authority path host port + + case "$endpoint" in + *://*) + scheme="${endpoint%%://*}" + rest="${endpoint#*://}" + ;; + *) + return 1 + ;; + esac + + authority="${rest%%/*}" + path="${rest#"$authority"}" + if [ "$path" = "$rest" ]; then + path="" + fi + + if [[ "$authority" =~ ^\[([^]]+)\]:(.+)$ ]]; then + host="${BASH_REMATCH[1]}" + port="${BASH_REMATCH[2]}" + elif [[ "$authority" =~ ^\[([^]]+)\]$ ]]; then + host="${BASH_REMATCH[1]}" + port="" + elif [[ "$authority" == *:* ]]; then + host="${authority%%:*}" + port="${authority##*:}" + else + host="$authority" + port="" + fi + + if [ -z "$port" ]; then + case "$scheme" in + https) port="443" ;; + *) port="80" ;; + esac + fi + + printf '%s\n%s\n%s\n%s\n' "$scheme" "$host" "$port" "$path" +} + +tcp_probe() { + local host="$1" + local port="$2" + + if command -v timeout >/dev/null 2>&1; then + timeout 2 bash -c "exec 3<>/dev/tcp/\$1/\$2" _ "$host" "$port" >/dev/null 2>&1 + else + bash -c "exec 3<>/dev/tcp/\$1/\$2" _ "$host" "$port" >/dev/null 2>&1 + fi +} + +ensure_host_gateway_aliases() { + # Seed /etc/hosts with the well-known gvproxy hostnames so the supervisor + # can reach the OpenShell server even when gvproxy's built-in DNS is not + # in resolv.conf (e.g. when DHCP fails and we fall back to 8.8.8.8). + # + # Critical distinction: host.* aliases point at the gvproxy *host-loopback* + # IP (192.168.127.254), not the gateway IP (192.168.127.1). Only the + # host-loopback IP carries NAT rewriting to the host's 127.0.0.1 — the + # gateway IP only listens on gvproxy's own service ports (DNS:53, DHCP, + # HTTP API:80). Pinning host.containers.internal to the gateway IP + # silently breaks guest→host port reachability for arbitrary ports. + local hosts_tmp="/tmp/openshell-hosts.$$" + local host_aliases="host.openshell.internal host.containers.internal host.docker.internal" + local gateway_aliases="gateway.containers.internal" + local filter='(^|[[:space:]])(host\.openshell\.internal|host\.containers\.internal|host\.docker\.internal|gateway\.containers\.internal)([[:space:]]|$)' + + if [ -f /etc/hosts ]; then + grep -vE "$filter" /etc/hosts > "$hosts_tmp" || true + else + : > "$hosts_tmp" + fi + + # In TAP/GPU mode, GATEWAY_IP is overridden to VM_NET_GW (the host-side + # of the TAP), and the gateway is reachable directly there. In gvproxy + # mode, host.openshell.internal etc. need GVPROXY_HOST_LOOPBACK_IP + # (192.168.127.254) which is gvproxy's host-NAT entry, while + # gateway.containers.internal points at the gvproxy gateway itself. + if [ "${GATEWAY_IP}" = "${GVPROXY_GATEWAY_IP}" ]; then + printf '%s %s\n' "$GVPROXY_HOST_LOOPBACK_IP" "$host_aliases" >> "$hosts_tmp" + printf '%s %s\n' "$GVPROXY_GATEWAY_IP" "$gateway_aliases" >> "$hosts_tmp" + else + # TAP networking: gateway and host are both reachable at GATEWAY_IP. + printf '%s %s %s\n' "$GATEWAY_IP" "$host_aliases" "$gateway_aliases" >> "$hosts_tmp" + fi + cat "$hosts_tmp" > /etc/hosts + rm -f "$hosts_tmp" +} + +rewrite_openshell_endpoint_if_needed() { + local endpoint="${OPENSHELL_ENDPOINT:-}" + [ -n "$endpoint" ] || return 0 + + local parsed + if ! parsed="$(parse_endpoint "$endpoint")"; then + ts "WARNING: could not parse OPENSHELL_ENDPOINT=$endpoint" + return 0 + fi + + local scheme host port path + scheme="$(printf '%s\n' "$parsed" | sed -n '1p')" + host="$(printf '%s\n' "$parsed" | sed -n '2p')" + port="$(printf '%s\n' "$parsed" | sed -n '3p')" + path="$(printf '%s\n' "$parsed" | sed -n '4p')" + + if tcp_probe "$host" "$port"; then + return 0 + fi + + # Probe candidates in preference order. Hostnames first for informative + # log output, then a bare IP as a final safety net. In gvproxy mode the + # bare IP is the host-loopback (192.168.127.254). In TAP/GPU mode it's + # the TAP host gateway. + local fallback_ip="$GVPROXY_HOST_LOOPBACK_IP" + if [ "${GATEWAY_IP}" != "${GVPROXY_GATEWAY_IP}" ]; then + fallback_ip="$GATEWAY_IP" + fi + for candidate in host.openshell.internal host.containers.internal host.docker.internal "$fallback_ip"; do + if [ "$candidate" = "$host" ]; then + continue + fi + if tcp_probe "$candidate" "$port"; then + local authority="$candidate" + if ! { [ "$scheme" = "http" ] && [ "$port" = "80" ]; } \ + && ! { [ "$scheme" = "https" ] && [ "$port" = "443" ]; }; then + authority="${authority}:${port}" + fi + export OPENSHELL_ENDPOINT="${scheme}://${authority}${path}" + ts "rewrote OPENSHELL_ENDPOINT to ${OPENSHELL_ENDPOINT}" + return 0 + fi + done + + ts "WARNING: could not reach OpenShell endpoint ${host}:${port}" +} + +create_gpu_device_nodes_mknod() { + # Mode 666 is intentional: single-tenant microVM with the VM itself as the + # isolation boundary. The sandbox user is the only non-root user. + local nv_major + nv_major=$(awk '$2 == "nvidia" {print $1}' /proc/devices 2>/dev/null || true) + if [ -n "$nv_major" ]; then + mknod -m 666 /dev/nvidiactl c "$nv_major" 255 2>/dev/null || true + + local gpu_count=0 + if [ -d /proc/driver/nvidia/gpus ]; then + for gpu_dir in /proc/driver/nvidia/gpus/*/; do + [ -d "$gpu_dir" ] || continue + mknod -m 666 "/dev/nvidia${gpu_count}" c "$nv_major" "$gpu_count" 2>/dev/null || true + gpu_count=$((gpu_count + 1)) + done + fi + if [ "$gpu_count" -eq 0 ]; then + mknod -m 666 /dev/nvidia0 c "$nv_major" 0 2>/dev/null || true + fi + + local modeset_major + modeset_major=$(awk '$2 == "nvidia-modeset" {print $1}' /proc/devices 2>/dev/null || true) + if [ -n "$modeset_major" ]; then + mknod -m 666 /dev/nvidia-modeset c "$modeset_major" 254 2>/dev/null || true + fi + + local uvm_major + uvm_major=$(awk '$2 == "nvidia-uvm" {print $1}' /proc/devices 2>/dev/null || true) + if [ -n "$uvm_major" ]; then + mknod -m 666 /dev/nvidia-uvm c "$uvm_major" 0 2>/dev/null || true + mknod -m 666 /dev/nvidia-uvm-tools c "$uvm_major" 1 2>/dev/null || true + fi + + local caps_major + caps_major=$(awk '$2 == "nvidia-caps" {print $1}' /proc/devices 2>/dev/null || true) + if [ -n "$caps_major" ]; then + mkdir -p /dev/nvidia-caps 2>/dev/null || true + mknod -m 666 /dev/nvidia-caps/nvidia-cap1 c "$caps_major" 1 2>/dev/null || true + mknod -m 666 /dev/nvidia-caps/nvidia-cap2 c "$caps_major" 2 2>/dev/null || true + fi + + ts "GPU device nodes created via mknod (${gpu_count} GPU(s), major=${nv_major})" + else + ts "WARNING: 'nvidia' not in /proc/devices; device nodes unavailable" + fi +} + +setup_gpu() { + ts "GPU_ENABLED=true — initializing GPU passthrough" + + if ! command -v modprobe >/dev/null 2>&1; then + ts "FATAL: modprobe not found; cannot load nvidia kernel modules" + return 1 + fi + + # Stage GSP firmware from virtiofs to tmpfs to avoid slow FUSE reads + if [ -d /lib/firmware/nvidia ]; then + ts "staging GPU firmware to tmpfs" + mkdir -p /run/firmware/nvidia + cp -a /lib/firmware/nvidia/* /run/firmware/nvidia/ 2>/dev/null || true + if [ -e /sys/module/firmware_class/parameters/path ]; then + echo /run/firmware > /sys/module/firmware_class/parameters/path + fi + fi + + ts "loading nvidia kernel modules" + modprobe nvidia || { ts "FATAL: modprobe nvidia failed"; return 1; } + modprobe nvidia_uvm 2>/dev/null || true + modprobe nvidia_modeset 2>/dev/null || true + + rm -rf /run/firmware 2>/dev/null || true + + if command -v nvidia-smi >/dev/null 2>&1; then + ts "running nvidia-smi to create device nodes and validate GPU" + local smi_rc=0 + nvidia-smi >/dev/null 2>&1 || smi_rc=$? + if [ "$smi_rc" -eq 0 ]; then + nvidia-smi -L 2>/dev/null | while read -r line; do ts " $line"; done + ts "GPU initialization successful" + else + ts "WARNING: nvidia-smi failed (exit ${smi_rc}); falling back to mknod" + create_gpu_device_nodes_mknod + fi + else + ts "nvidia-smi not found; creating device nodes via mknod" + create_gpu_device_nodes_mknod + fi +} + +mount -t proc proc /proc 2>/dev/null & +mount -t sysfs sysfs /sys 2>/dev/null & +mount -t tmpfs tmpfs /tmp 2>/dev/null & +mount -t tmpfs tmpfs /run 2>/dev/null & +mount -t devtmpfs devtmpfs /dev 2>/dev/null & +wait + +mkdir -p /dev/pts /dev/shm /sys/fs/cgroup +mount -t devpts devpts /dev/pts 2>/dev/null & +mount -t tmpfs tmpfs /dev/shm 2>/dev/null & +mount -t cgroup2 cgroup2 /sys/fs/cgroup 2>/dev/null & +wait + +hostname openshell-sandbox-vm 2>/dev/null || true +ip link set lo up 2>/dev/null || true + +# GPU initialization (before networking so nvidia-smi output is visible early) +if [ "${GPU_ENABLED}" = "true" ]; then + setup_gpu || ts "WARNING: GPU init failed; continuing without GPU" +fi + +# Networking: use TAP static config if VM_NET_IP is set (QEMU path), +# otherwise fall back to gvproxy DHCP on eth0 (libkrun path). +if [ -n "${VM_NET_IP}" ] && [ -n "${VM_NET_GW}" ]; then + ts "configuring TAP networking (static ${VM_NET_IP} gw ${VM_NET_GW})" + GATEWAY_IP="${VM_NET_GW}" + + TAP_NIC="" + NIC_WAIT=0 + while [ -z "$TAP_NIC" ] && [ "$NIC_WAIT" -lt 10 ]; do + for candidate in eth0 ens3 enp0s2; do + if ip link show "$candidate" >/dev/null 2>&1 && [ "$candidate" != "lo" ]; then + TAP_NIC="$candidate" + break + fi + done + if [ -z "$TAP_NIC" ]; then + for sys_nic in /sys/class/net/*; do + [ -e "$sys_nic" ] || continue + candidate="${sys_nic##*/}" + if ip link show "$candidate" >/dev/null 2>&1 && [ "$candidate" != "lo" ]; then + TAP_NIC="$candidate" + break + fi + done + fi + if [ -z "$TAP_NIC" ]; then + sleep 1 + NIC_WAIT=$((NIC_WAIT + 1)) + fi + done + + if [ -n "$TAP_NIC" ]; then + ts "using NIC ${TAP_NIC} for TAP networking" + ip link set "$TAP_NIC" up 2>/dev/null || true + ip addr add "${VM_NET_IP}/30" dev "$TAP_NIC" 2>/dev/null || true + ip route add default via "${VM_NET_GW}" 2>/dev/null || true + else + ts "WARNING: no network interface found for TAP networking" + fi + + if [ -n "${VM_NET_DNS}" ]; then + echo "nameserver ${VM_NET_DNS}" > /etc/resolv.conf + elif [ ! -s /etc/resolv.conf ]; then + echo "nameserver 8.8.8.8" > /etc/resolv.conf + echo "nameserver 8.8.4.4" >> /etc/resolv.conf + fi + + ensure_host_gateway_aliases +elif ip link show eth0 >/dev/null 2>&1; then + ts "detected eth0 (gvproxy networking)" + ip link set eth0 up 2>/dev/null || true + + if command -v udhcpc >/dev/null 2>&1; then + UDHCPC_SCRIPT="/usr/share/udhcpc/default.script" + if [ ! -f "$UDHCPC_SCRIPT" ]; then + UDHCPC_SCRIPT="/run/openshell-udhcpc.script" + cat > "$UDHCPC_SCRIPT" <<'DHCP_SCRIPT' +#!/bin/sh +case "$1" in + bound|renew) + ip addr flush dev "$interface" + ip addr add "$ip/$mask" dev "$interface" + if [ -n "$router" ]; then + ip route add default via "$router" dev "$interface" + fi + if [ -n "$dns" ]; then + : > /etc/resolv.conf + for d in $dns; do + echo "nameserver $d" >> /etc/resolv.conf + done + fi + ;; +esac +DHCP_SCRIPT + chmod +x "$UDHCPC_SCRIPT" + fi + + if ! udhcpc -i eth0 -f -q -n -T 1 -t 3 -A 1 -s "$UDHCPC_SCRIPT" 2>&1; then + ts "WARNING: DHCP failed, falling back to static config" + ip addr add 192.168.127.2/24 dev eth0 2>/dev/null || true + ip route add default via "$GVPROXY_GATEWAY_IP" 2>/dev/null || true + fi + else + ts "no DHCP client, using static config" + ip addr add 192.168.127.2/24 dev eth0 2>/dev/null || true + ip route add default via "$GVPROXY_GATEWAY_IP" 2>/dev/null || true + fi + + if [ ! -s /etc/resolv.conf ]; then + echo "nameserver 8.8.8.8" > /etc/resolv.conf + echo "nameserver 8.8.4.4" >> /etc/resolv.conf + fi + + ensure_host_gateway_aliases +else + ts "WARNING: no network interface found; supervisor will start without guest egress" +fi + +export HOME=/sandbox +export USER=sandbox + +rewrite_openshell_endpoint_if_needed + +# Log supervisor connectivity state for debugging stuck-in-Provisioning issues +if [ -n "${OPENSHELL_ENDPOINT:-}" ]; then + _ep_parsed="$(parse_endpoint "$OPENSHELL_ENDPOINT" 2>/dev/null || true)" + if [ -n "$_ep_parsed" ]; then + _ep_host="$(printf '%s\n' "$_ep_parsed" | sed -n '2p')" + _ep_port="$(printf '%s\n' "$_ep_parsed" | sed -n '3p')" + if tcp_probe "$_ep_host" "$_ep_port"; then + ts "gateway reachable at ${_ep_host}:${_ep_port}" + else + ts "WARNING: gateway NOT reachable at ${_ep_host}:${_ep_port} — supervisor may fail to connect" + fi + fi + ts "OPENSHELL_ENDPOINT=${OPENSHELL_ENDPOINT}" +fi +if [ -n "${OPENSHELL_SANDBOX_ID:-}" ]; then + ts "OPENSHELL_SANDBOX_ID=${OPENSHELL_SANDBOX_ID}" +fi + +ts "starting openshell-sandbox supervisor" +exec /opt/openshell/bin/openshell-sandbox --workdir /sandbox diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs new file mode 100644 index 0000000000..d79e5d922a --- /dev/null +++ b/crates/openshell-driver-vm/src/driver.rs @@ -0,0 +1,3208 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use crate::gpu::{ + GpuInventory, SubnetAllocator, allocate_vsock_cid, mac_from_sandbox_id, tap_device_name, +}; +use crate::rootfs::{ + create_rootfs_archive_from_dir, extract_rootfs_archive_to, + prepare_sandbox_rootfs_from_image_root, sandbox_guest_init_path, +}; +use bollard::Docker; +use bollard::errors::Error as BollardError; +use bollard::models::ContainerCreateBody; +use bollard::query_parameters::{CreateContainerOptionsBuilder, RemoveContainerOptionsBuilder}; +use flate2::read::GzDecoder; +use futures::{Stream, StreamExt}; +use nix::errno::Errno; +use nix::sys::signal::{Signal, kill}; +use nix::unistd::Pid; +use oci_client::client::{Client as OciClient, ClientConfig}; +use oci_client::manifest::{ImageIndexEntry, OciDescriptor}; +use oci_client::secrets::RegistryAuth; +use oci_client::{Reference, RegistryOperation}; +use openshell_core::proto::compute::v1::{ + CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, + DriverCondition as SandboxCondition, DriverPlatformEvent as PlatformEvent, + DriverSandbox as Sandbox, DriverSandboxStatus as SandboxStatus, GetCapabilitiesRequest, + GetCapabilitiesResponse, GetSandboxRequest, GetSandboxResponse, ListSandboxesRequest, + ListSandboxesResponse, StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateRequest, + ValidateSandboxCreateResponse, WatchSandboxesDeletedEvent, WatchSandboxesEvent, + WatchSandboxesPlatformEvent, WatchSandboxesRequest, WatchSandboxesSandboxEvent, + compute_driver_server::ComputeDriver, watch_sandboxes_event, +}; +use openshell_vfio::SysfsRoot; +use sha2::{Digest, Sha256}; +use std::collections::{HashMap, HashSet}; +use std::fs; +use std::io::Read; +use std::net::Ipv4Addr; +use std::os::unix::fs::PermissionsExt; +use std::path::{Path, PathBuf}; +use std::pin::Pin; +use std::process::Stdio; +use std::sync::Arc; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; +use tokio::io::AsyncWriteExt; +use tokio::process::{Child, Command}; +use tokio::sync::{Mutex, broadcast, mpsc}; +use tokio_stream::wrappers::ReceiverStream; +use tonic::{Request, Response, Status}; +use tracing::{info, warn}; +use url::{Host, Url}; + +const DRIVER_NAME: &str = "openshell-driver-vm"; +const WATCH_BUFFER: usize = 256; +const DEFAULT_VCPUS: u8 = 2; +const DEFAULT_MEM_MIB: u32 = 2048; +/// gvproxy host-loopback IP — gvproxy's TCP/UDP/ICMP forwarder NAT-rewrites +/// this destination to the host's `127.0.0.1` and dials out from the host +/// process. This is the only address that transparently reaches host-bound +/// services without explicit `expose` rules. +/// +/// See gvisor-tap-vsock `cmd/gvproxy/config.go` (default NAT entry +/// `HostIP -> 127.0.0.1`) and `pkg/services/forwarder/tcp.go` (NAT lookup +/// before `net.Dial`). +/// +/// Code paths route via `GVPROXY_HOST_LOOPBACK_ALIAS` (DNS / /etc/hosts) +/// instead so logs stay readable; this constant is kept for documentation +/// and parity with the guest init script. +#[allow(dead_code)] +const GVPROXY_HOST_LOOPBACK_IP: &str = "192.168.127.254"; +const OPENSHELL_HOST_GATEWAY_ALIAS: &str = "host.openshell.internal"; +/// Hostname gvproxy resolves (via its embedded DNS) to the host-loopback IP. +/// +/// We rewrite loopback URLs to this hostname rather than the bare IP because: +/// * the guest init script seeds /etc/hosts with the same mapping, so it +/// resolves even when gvproxy's DNS is not in resolv.conf; +/// * keeping a recognisable hostname makes log messages clearer than a bare +/// 192.168.127.254 reference; +/// * `host.docker.internal` works the same way for Docker-flavoured tooling. +/// +/// Both names ultimately route through the gvproxy NAT path on +/// `GVPROXY_HOST_LOOPBACK_IP` — they do **not** go through the gateway IP. +const GVPROXY_HOST_LOOPBACK_ALIAS: &str = "host.containers.internal"; +const GUEST_SSH_SOCKET_PATH: &str = "/run/openshell/ssh.sock"; +const GUEST_TLS_DIR: &str = "/opt/openshell/tls"; +const GUEST_TLS_CA_PATH: &str = "/opt/openshell/tls/ca.crt"; +const GUEST_TLS_CERT_PATH: &str = "/opt/openshell/tls/tls.crt"; +const GUEST_TLS_KEY_PATH: &str = "/opt/openshell/tls/tls.key"; +const IMAGE_CACHE_ROOT_DIR: &str = "images"; +const IMAGE_CACHE_ROOTFS_ARCHIVE: &str = "rootfs.tar"; +const IMAGE_EXPORT_ROOTFS_ARCHIVE: &str = "source-rootfs.tar"; +const IMAGE_IDENTITY_FILE: &str = "image-identity"; +const IMAGE_REFERENCE_FILE: &str = "image-reference"; +static IMAGE_CACHE_BUILD_COUNTER: AtomicU64 = AtomicU64::new(0); + +#[derive(Debug, Clone)] +struct VmDriverTlsPaths { + ca: PathBuf, + cert: PathBuf, + key: PathBuf, +} + +#[derive(Debug, Clone)] +pub struct VmDriverConfig { + pub openshell_endpoint: String, + pub state_dir: PathBuf, + pub launcher_bin: Option, + pub default_image: String, + pub ssh_handshake_secret: String, + pub ssh_handshake_skew_secs: u64, + pub log_level: String, + pub krun_log_level: u32, + pub vcpus: u8, + pub mem_mib: u32, + pub guest_tls_ca: Option, + pub guest_tls_cert: Option, + pub guest_tls_key: Option, + pub gpu_enabled: bool, + pub gpu_mem_mib: u32, + pub gpu_vcpus: u8, +} + +impl Default for VmDriverConfig { + fn default() -> Self { + Self { + openshell_endpoint: String::new(), + state_dir: PathBuf::from("target/openshell-vm-driver"), + launcher_bin: None, + default_image: String::new(), + ssh_handshake_secret: String::new(), + ssh_handshake_skew_secs: 300, + log_level: "info".to_string(), + krun_log_level: 1, + vcpus: DEFAULT_VCPUS, + mem_mib: DEFAULT_MEM_MIB, + guest_tls_ca: None, + guest_tls_cert: None, + guest_tls_key: None, + gpu_enabled: false, + gpu_mem_mib: 8192, + gpu_vcpus: 4, + } + } +} + +impl VmDriverConfig { + fn requires_tls_materials(&self) -> bool { + self.openshell_endpoint.starts_with("https://") + } + + fn tls_paths(&self) -> Result, String> { + let provided = [ + self.guest_tls_ca.as_ref(), + self.guest_tls_cert.as_ref(), + self.guest_tls_key.as_ref(), + ]; + if provided.iter().all(Option::is_none) { + return if self.requires_tls_materials() { + Err( + "https:// openshell endpoint requires OPENSHELL_VM_TLS_CA, OPENSHELL_VM_TLS_CERT, and OPENSHELL_VM_TLS_KEY so sandbox VMs can authenticate to the gateway" + .to_string(), + ) + } else { + Ok(None) + }; + } + + let Some(ca) = self.guest_tls_ca.clone() else { + return Err( + "OPENSHELL_VM_TLS_CA is required when TLS materials are configured".to_string(), + ); + }; + let Some(cert) = self.guest_tls_cert.clone() else { + return Err( + "OPENSHELL_VM_TLS_CERT is required when TLS materials are configured".to_string(), + ); + }; + let Some(key) = self.guest_tls_key.clone() else { + return Err( + "OPENSHELL_VM_TLS_KEY is required when TLS materials are configured".to_string(), + ); + }; + + for path in [&ca, &cert, &key] { + if !path.is_file() { + return Err(format!( + "TLS material '{}' does not exist or is not a file", + path.display() + )); + } + } + + Ok(Some(VmDriverTlsPaths { ca, cert, key })) + } +} + +fn validate_openshell_endpoint(endpoint: &str) -> Result<(), String> { + let url = Url::parse(endpoint) + .map_err(|err| format!("invalid openshell endpoint '{endpoint}': {err}"))?; + let Some(host) = url.host() else { + return Err(format!("openshell endpoint '{endpoint}' is missing a host")); + }; + + let invalid_from_vm = match host { + Host::Domain(_) => false, + Host::Ipv4(ip) => ip.is_unspecified(), + Host::Ipv6(ip) => ip.is_unspecified(), + }; + + if invalid_from_vm { + return Err(format!( + "openshell endpoint '{endpoint}' is not reachable from sandbox VMs; use a concrete host such as 127.0.0.1, {OPENSHELL_HOST_GATEWAY_ALIAS}, or another routable address" + )); + } + + Ok(()) +} + +#[derive(Debug)] +struct VmProcess { + child: Child, + deleting: bool, +} + +#[derive(Debug)] +struct SandboxRecord { + snapshot: Sandbox, + state_dir: PathBuf, + process: Arc>, + gpu_bdf: Option, +} + +#[derive(Clone)] +pub struct VmDriver { + config: VmDriverConfig, + launcher_bin: PathBuf, + registry: Arc>>, + image_cache_lock: Arc>, + events: broadcast::Sender, + gpu_inventory: Option>>, + subnet_allocator: Arc>, +} + +impl VmDriver { + pub async fn new(config: VmDriverConfig) -> Result { + if config.openshell_endpoint.trim().is_empty() { + return Err("openshell endpoint is required".to_string()); + } + validate_openshell_endpoint(&config.openshell_endpoint)?; + let _ = config.tls_paths()?; + + #[cfg(target_os = "linux")] + if config.gpu_enabled { + check_gpu_privileges()?; + tokio::task::spawn_blocking(crate::cleanup_stale_tap_interfaces) + .await + .map_err(|e| format!("cleanup stale TAP interfaces panicked: {e}"))?; + } + + let state_root = sandboxes_root_dir(&config.state_dir); + tokio::fs::create_dir_all(&state_root) + .await + .map_err(|err| { + format!( + "failed to create state dir '{}': {err}", + state_root.display() + ) + })?; + let image_cache_root = image_cache_root_dir(&config.state_dir); + tokio::fs::create_dir_all(&image_cache_root) + .await + .map_err(|err| { + format!( + "failed to create state dir '{}': {err}", + image_cache_root.display() + ) + })?; + + let launcher_bin = if let Some(path) = config.launcher_bin.clone() { + path + } else { + std::env::current_exe() + .map_err(|err| format!("failed to resolve vm driver executable: {err}"))? + }; + + let gpu_inventory = if config.gpu_enabled { + let sysfs = SysfsRoot::system(); + let inventory = GpuInventory::new(sysfs, &config.state_dir); + tracing::info!( + gpu_count = inventory.gpu_count(), + "GPU inventory initialized" + ); + Some(Arc::new(std::sync::Mutex::new(inventory))) + } else { + None + }; + + let subnet_allocator = Arc::new(std::sync::Mutex::new(SubnetAllocator::new( + Ipv4Addr::new(10, 0, 128, 0), + 17, + ))); + + let (events, _) = broadcast::channel(WATCH_BUFFER); + Ok(Self { + config, + launcher_bin, + registry: Arc::new(Mutex::new(HashMap::new())), + image_cache_lock: Arc::new(Mutex::new(())), + events, + gpu_inventory, + subnet_allocator, + }) + } + + #[must_use] + pub fn capabilities(&self) -> GetCapabilitiesResponse { + let gpu_count = self + .gpu_inventory + .as_ref() + .and_then(|inv| inv.lock().ok()) + .map_or(0, |inv| inv.gpu_count()); + GetCapabilitiesResponse { + driver_name: DRIVER_NAME.to_string(), + driver_version: openshell_core::VERSION.to_string(), + default_image: self.config.default_image.clone(), + supports_gpu: self.gpu_inventory.is_some(), + gpu_count, + } + } + + // `tonic::Status` is large but is the standard error type across the + // gRPC API surface; boxing here would diverge from every other handler. + #[allow(clippy::result_large_err)] + pub fn validate_sandbox(&self, sandbox: &Sandbox) -> Result<(), Status> { + validate_vm_sandbox(sandbox, self.config.gpu_enabled)?; + if self.resolved_sandbox_image(sandbox).is_none() { + return Err(Status::failed_precondition( + "vm sandboxes require template.image or a configured default sandbox image", + )); + } + Ok(()) + } + + // `tonic::Status` is large but is the standard error type across the + // gRPC API surface; boxing here would diverge from every other handler. + #[allow(clippy::result_large_err)] + pub async fn create_sandbox(&self, sandbox: &Sandbox) -> Result { + info!( + sandbox_id = %sandbox.id, + sandbox_name = %sandbox.name, + "vm driver: create_sandbox received" + ); + validate_vm_sandbox(sandbox, self.config.gpu_enabled)?; + + if self.registry.lock().await.contains_key(&sandbox.id) { + return Err(Status::already_exists("sandbox already exists")); + } + + let spec = sandbox.spec.as_ref(); + let is_gpu = spec.is_some_and(|s| s.gpu); + let gpu_device = spec.map_or("", |s| s.gpu_device.as_str()); + + let state_dir = sandbox_state_dir(&self.config.state_dir, &sandbox.id); + let rootfs = state_dir.join("rootfs"); + let image_ref = self.resolved_sandbox_image(sandbox).ok_or_else(|| { + Status::failed_precondition( + "vm sandboxes require template.image or a configured default sandbox image", + ) + })?; + info!( + sandbox_id = %sandbox.id, + image_ref = %image_ref, + state_dir = %state_dir.display(), + "vm driver: resolved image ref, preparing rootfs" + ); + + tokio::fs::create_dir_all(&state_dir) + .await + .map_err(|err| Status::internal(format!("create state dir failed: {err}")))?; + + let tls_paths = self + .config + .tls_paths() + .map_err(Status::failed_precondition)?; + // Mirror the K8s `Scheduled` event so the CLI can complete the + // "Requesting sandbox" step and switch the spinner over to the + // image-pull phase before we block on the registry. + self.publish_platform_event( + sandbox.id.clone(), + platform_event( + "vm", + "Normal", + "Scheduled", + format!("Sandbox accepted by vm driver to image \"{image_ref}\""), + ), + ); + + let image_identity = match self + .prepare_runtime_rootfs(&sandbox.id, &image_ref, &rootfs) + .await + { + Ok(image_identity) => { + info!( + sandbox_id = %sandbox.id, + image_identity = %image_identity, + "vm driver: rootfs prepared" + ); + image_identity + } + Err(err) => { + warn!( + sandbox_id = %sandbox.id, + error = %err.message(), + "vm driver: rootfs preparation failed" + ); + let _ = tokio::fs::remove_dir_all(&state_dir).await; + return Err(err); + } + }; + if let Some(tls_paths) = tls_paths.as_ref() + && let Err(err) = prepare_guest_tls_materials(&rootfs, tls_paths).await + { + let _ = tokio::fs::remove_dir_all(&state_dir).await; + return Err(Status::internal(format!( + "prepare guest TLS materials failed: {err}" + ))); + } + + if let Err(err) = + write_sandbox_image_metadata(&state_dir, &image_ref, &image_identity).await + { + let _ = tokio::fs::remove_dir_all(&state_dir).await; + return Err(Status::internal(format!( + "write sandbox image metadata failed: {err}" + ))); + } + + let gpu_bdf = if is_gpu { + let inventory = self + .gpu_inventory + .as_ref() + .ok_or_else(|| Status::internal("GPU inventory not initialized"))?; + match inventory + .lock() + .map_err(|e| Status::internal(format!("GPU inventory lock poisoned: {e}"))) + .and_then(|mut inv| { + inv.assign(&sandbox.id, gpu_device) + .map_err(Status::failed_precondition) + }) { + Ok(assignment) => { + tracing::info!( + sandbox_id = %sandbox.id, + bdf = %assignment.bdf, + gpu_name = %assignment.name, + iommu_group = assignment.iommu_group, + "assigned GPU to sandbox" + ); + Some(assignment.bdf) + } + Err(err) => { + let _ = tokio::fs::remove_dir_all(&state_dir).await; + return Err(err); + } + } + } else { + None + }; + + let console_output = state_dir.join("rootfs-console.log"); + let mut command = Command::new(&self.launcher_bin); + command.stdin(Stdio::null()); + command.stdout(Stdio::inherit()); + command.stderr(Stdio::inherit()); + command.arg("--internal-run-vm"); + command.arg("--vm-rootfs").arg(&rootfs); + command.arg("--vm-exec").arg(sandbox_guest_init_path()); + command.arg("--vm-workdir").arg("/"); + command.arg("--vm-console-output").arg(&console_output); + + // Compute the endpoint override before building the env so + // there is a single OPENSHELL_ENDPOINT value in the env list. + let endpoint_override = if let Some(bdf) = gpu_bdf.as_ref() { + let subnet = match self + .subnet_allocator + .lock() + .map_err(|e| Status::internal(format!("subnet allocator lock poisoned: {e}"))) + .and_then(|mut alloc| { + alloc + .allocate(&sandbox.id) + .map_err(Status::failed_precondition) + }) { + Ok(s) => s, + Err(err) => { + self.release_gpu_and_subnet(&sandbox.id); + let _ = tokio::fs::remove_dir_all(&state_dir).await; + return Err(err); + } + }; + let vsock_cid = allocate_vsock_cid(); + let mac = mac_from_sandbox_id(&sandbox.id); + let mac_str = format!( + "{:02x}:{:02x}:{:02x}:{:02x}:{:02x}:{:02x}", + mac[0], mac[1], mac[2], mac[3], mac[4], mac[5] + ); + let tap = tap_device_name(&sandbox.id); + + let tap_endpoint = guest_visible_openshell_endpoint_for_tap( + &self.config.openshell_endpoint, + &subnet.host_ip.to_string(), + ); + + command.arg("--vm-backend").arg("qemu"); + command + .arg("--vm-vcpus") + .arg(self.config.gpu_vcpus.to_string()); + command + .arg("--vm-mem-mib") + .arg(self.config.gpu_mem_mib.to_string()); + command.arg("--vm-gpu-bdf").arg(bdf); + command.arg("--vm-tap-device").arg(&tap); + command + .arg("--vm-guest-ip") + .arg(subnet.guest_ip.to_string()); + command.arg("--vm-host-ip").arg(subnet.host_ip.to_string()); + command.arg("--vm-vsock-cid").arg(vsock_cid.to_string()); + command.arg("--vm-guest-mac").arg(&mac_str); + + if let Some(port) = gateway_port_from_endpoint(&self.config.openshell_endpoint) { + command.arg("--vm-gateway-port").arg(port.to_string()); + } + + Some(tap_endpoint) + } else { + command.arg("--vm-vcpus").arg(self.config.vcpus.to_string()); + command + .arg("--vm-mem-mib") + .arg(self.config.mem_mib.to_string()); + None + }; + + command + .arg("--vm-krun-log-level") + .arg(self.config.krun_log_level.to_string()); + + for env in build_guest_environment(sandbox, &self.config, endpoint_override.as_deref()) { + command.arg("--vm-env").arg(env); + } + + info!( + sandbox_id = %sandbox.id, + launcher = %self.launcher_bin.display(), + console_output = %console_output.display(), + "vm driver: spawning VM launcher" + ); + let child = match command.spawn() { + Ok(child) => child, + Err(err) => { + warn!( + sandbox_id = %sandbox.id, + error = %err, + "vm driver: launcher spawn failed" + ); + if gpu_bdf.is_some() { + self.release_gpu_and_subnet(&sandbox.id); + } + let _ = tokio::fs::remove_dir_all(&state_dir).await; + return Err(Status::internal(format!( + "failed to launch vm helper '{}': {err}", + self.launcher_bin.display() + ))); + } + }; + info!( + sandbox_id = %sandbox.id, + launcher_pid = child.id().unwrap_or(0), + "vm driver: launcher spawned" + ); + // Mirror the K8s `Started` event so the CLI can complete the + // "Starting sandbox" step. The supervisor-ready transition still + // promotes the sandbox to `Ready` separately. + self.publish_platform_event( + sandbox.id.clone(), + platform_event("vm", "Normal", "Started", "Started VM launcher".to_string()), + ); + let snapshot = sandbox_snapshot(sandbox, provisioning_condition(), false); + let process = Arc::new(Mutex::new(VmProcess { + child, + deleting: false, + })); + + { + let mut registry = self.registry.lock().await; + registry.insert( + sandbox.id.clone(), + SandboxRecord { + snapshot: snapshot.clone(), + state_dir: state_dir.clone(), + process: process.clone(), + gpu_bdf: gpu_bdf.clone(), + }, + ); + } + + self.publish_snapshot(snapshot.clone()); + tokio::spawn({ + let driver = self.clone(); + let sandbox_id = sandbox.id.clone(); + async move { + driver.monitor_sandbox(sandbox_id).await; + } + }); + + Ok(CreateSandboxResponse {}) + } + + pub async fn delete_sandbox( + &self, + sandbox_id: &str, + sandbox_name: &str, + ) -> Result { + let record = { + let registry = self.registry.lock().await; + if let Some((id, record)) = registry.get_key_value(sandbox_id) { + Some(( + id.clone(), + record.state_dir.clone(), + record.process.clone(), + record.gpu_bdf.clone(), + )) + } else { + let matched_id = registry + .iter() + .find(|(_, record)| record.snapshot.name == sandbox_name) + .map(|(id, _)| id.clone()); + matched_id.and_then(|id| { + registry.get(&id).map(|record| { + ( + id, + record.state_dir.clone(), + record.process.clone(), + record.gpu_bdf.clone(), + ) + }) + }) + } + }; + + let Some((record_id, state_dir, process, gpu_bdf)) = record else { + return Ok(DeleteSandboxResponse { deleted: false }); + }; + + if let Some(snapshot) = self + .set_snapshot_condition(&record_id, deleting_condition(), true) + .await + { + self.publish_snapshot(snapshot); + } + + { + let mut process = process.lock().await; + process.deleting = true; + terminate_vm_process(&mut process.child) + .await + .map_err(|err| Status::internal(format!("failed to stop vm: {err}")))?; + } + + if gpu_bdf.is_some() { + self.release_gpu_and_subnet(&record_id); + } + + if let Err(err) = tokio::fs::remove_dir_all(&state_dir).await + && err.kind() != std::io::ErrorKind::NotFound + { + return Err(Status::internal(format!( + "failed to remove state dir: {err}" + ))); + } + + { + let mut registry = self.registry.lock().await; + registry.remove(&record_id); + } + + self.publish_deleted(record_id); + Ok(DeleteSandboxResponse { deleted: true }) + } + + pub async fn get_sandbox( + &self, + sandbox_id: &str, + sandbox_name: &str, + ) -> Result, Status> { + let registry = self.registry.lock().await; + let sandbox = if sandbox_id.is_empty() { + registry + .values() + .find(|record| record.snapshot.name == sandbox_name) + .map(|record| record.snapshot.clone()) + } else { + registry + .get(sandbox_id) + .map(|record| record.snapshot.clone()) + }; + Ok(sandbox) + } + + pub async fn current_snapshots(&self) -> Vec { + let registry = self.registry.lock().await; + let mut snapshots = registry + .values() + .map(|record| record.snapshot.clone()) + .collect::>(); + snapshots.sort_by(|left, right| left.name.cmp(&right.name)); + snapshots + } + + fn release_gpu_and_subnet(&self, sandbox_id: &str) { + if let Some(inventory) = self.gpu_inventory.as_ref() + && let Ok(mut inv) = inventory.lock() + { + inv.release(sandbox_id); + } + if let Ok(mut alloc) = self.subnet_allocator.lock() { + alloc.release(sandbox_id); + } + } + + async fn prepare_runtime_rootfs( + &self, + sandbox_id: &str, + image_ref: &str, + rootfs: &Path, + ) -> Result { + let image_identity = self + .ensure_cached_image_rootfs_archive(sandbox_id, image_ref) + .await?; + let archive_path = image_cache_rootfs_archive(&self.config.state_dir, &image_identity); + let rootfs_dest = rootfs.to_path_buf(); + tokio::task::spawn_blocking(move || extract_rootfs_archive_to(&archive_path, &rootfs_dest)) + .await + .map_err(|err| Status::internal(format!("sandbox rootfs extraction panicked: {err}")))? + .map_err(|err| Status::internal(format!("extract sandbox rootfs failed: {err}")))?; + + Ok(image_identity) + } + + fn resolved_sandbox_image(&self, sandbox: &Sandbox) -> Option { + requested_sandbox_image(sandbox) + .map(ToOwned::to_owned) + .or_else(|| { + let image = self.config.default_image.trim(); + (!image.is_empty()).then(|| image.to_string()) + }) + } + + async fn ensure_cached_image_rootfs_archive( + &self, + sandbox_id: &str, + image_ref: &str, + ) -> Result { + if let Some((docker, image_identity)) = self.resolve_local_docker_image(image_ref).await? { + return self + .ensure_cached_local_image_rootfs_archive( + sandbox_id, + image_ref, + &docker, + &image_identity, + ) + .await; + } + + info!(image_ref = %image_ref, "vm driver: ensuring cached image rootfs archive (registry)"); + let reference = parse_registry_reference(image_ref)?; + let client = registry_client(); + let auth = registry_auth(image_ref)?; + info!(image_ref = %image_ref, "vm driver: authenticating with registry"); + client + .auth(&reference, &auth, RegistryOperation::Pull) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to authenticate registry access for vm sandbox image '{image_ref}': {err}" + )) + })?; + info!(image_ref = %image_ref, "vm driver: fetching manifest digest"); + let image_identity = client + .fetch_manifest_digest(&reference, &auth) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to resolve vm sandbox image '{image_ref}': {err}" + )) + })?; + info!( + image_ref = %image_ref, + image_identity = %image_identity, + "vm driver: manifest digest resolved" + ); + let archive_path = image_cache_rootfs_archive(&self.config.state_dir, &image_identity); + + // Mirror the K8s `Pulling` event so the CLI flips to the + // image-pull spinner with the image name as detail. We emit it + // for cache hits too and immediately follow with `Pulled` so the + // spinner step still advances cleanly. + self.publish_platform_event( + sandbox_id.to_string(), + platform_event( + "vm", + "Normal", + "Pulling", + format!("Pulling image \"{image_ref}\""), + ), + ); + + if tokio::fs::metadata(&archive_path).await.is_ok() { + info!( + image_identity = %image_identity, + archive_path = %archive_path.display(), + "vm driver: image rootfs archive cache hit (no build needed)" + ); + self.publish_pulled_event(sandbox_id, image_ref, &archive_path) + .await; + return Ok(image_identity); + } + + info!( + image_identity = %image_identity, + "vm driver: image rootfs archive cache miss, acquiring build lock" + ); + let _cache_guard = self.image_cache_lock.lock().await; + info!( + image_identity = %image_identity, + "vm driver: build lock acquired" + ); + if tokio::fs::metadata(&archive_path).await.is_ok() { + info!( + image_identity = %image_identity, + "vm driver: image rootfs archive cache hit after lock (built by another task)" + ); + self.publish_pulled_event(sandbox_id, image_ref, &archive_path) + .await; + return Ok(image_identity); + } + + self.build_cached_registry_image_rootfs_archive( + sandbox_id, + &client, + &reference, + &auth, + image_ref, + &image_identity, + ) + .await?; + self.publish_pulled_event(sandbox_id, image_ref, &archive_path) + .await; + Ok(image_identity) + } + + async fn resolve_local_docker_image( + &self, + image_ref: &str, + ) -> Result, Status> { + let required_local_image = is_openshell_local_build_image_ref(image_ref); + let docker = match Docker::connect_with_local_defaults() { + Ok(docker) => docker, + Err(err) if required_local_image => { + return Err(Status::failed_precondition(format!( + "failed to connect to local Docker daemon for locally built sandbox image '{image_ref}': {err}" + ))); + } + Err(err) => { + warn!( + image_ref = %image_ref, + error = %err, + "vm driver: local Docker daemon unavailable, falling back to registry" + ); + return Ok(None); + } + }; + + match docker.inspect_image(image_ref).await { + Ok(inspect) => { + if let Some(message) = local_docker_image_platform_mismatch( + image_ref, + inspect.os.as_deref(), + inspect.architecture.as_deref(), + ) { + if required_local_image { + return Err(Status::failed_precondition(message)); + } + warn!( + image_ref = %image_ref, + %message, + "vm driver: local Docker image platform mismatch, falling back to registry" + ); + return Ok(None); + } + + let image_identity = + inspect + .id + .filter(|id| !id.trim().is_empty()) + .ok_or_else(|| { + Status::failed_precondition(format!( + "local Docker image '{image_ref}' inspect response has no image ID" + )) + })?; + info!( + image_ref = %image_ref, + image_identity = %image_identity, + "vm driver: resolved image from local Docker daemon" + ); + Ok(Some((docker, image_identity))) + } + Err(err) if is_docker_not_found_error(&err) && required_local_image => { + Err(Status::failed_precondition(format!( + "locally built sandbox image '{image_ref}' is not present in the local Docker daemon" + ))) + } + Err(err) if is_docker_not_found_error(&err) => Ok(None), + Err(err) if required_local_image => Err(Status::failed_precondition(format!( + "failed to inspect locally built sandbox image '{image_ref}': {err}" + ))), + Err(err) => { + warn!( + image_ref = %image_ref, + error = %err, + "vm driver: local Docker image inspection failed, falling back to registry" + ); + Ok(None) + } + } + } + + async fn ensure_cached_local_image_rootfs_archive( + &self, + sandbox_id: &str, + image_ref: &str, + docker: &Docker, + image_identity: &str, + ) -> Result { + let archive_path = image_cache_rootfs_archive(&self.config.state_dir, image_identity); + + self.publish_platform_event( + sandbox_id.to_string(), + platform_event( + "vm", + "Normal", + "Pulling", + format!("Pulling image \"{image_ref}\""), + ), + ); + + if tokio::fs::metadata(&archive_path).await.is_ok() { + self.publish_pulled_event(sandbox_id, image_ref, &archive_path) + .await; + return Ok(image_identity.to_string()); + } + + let _cache_guard = self.image_cache_lock.lock().await; + if tokio::fs::metadata(&archive_path).await.is_ok() { + self.publish_pulled_event(sandbox_id, image_ref, &archive_path) + .await; + return Ok(image_identity.to_string()); + } + + self.build_cached_local_image_rootfs_archive(docker, image_ref, image_identity) + .await?; + self.publish_pulled_event(sandbox_id, image_ref, &archive_path) + .await; + Ok(image_identity.to_string()) + } + + async fn build_cached_local_image_rootfs_archive( + &self, + docker: &Docker, + image_ref: &str, + image_identity: &str, + ) -> Result<(), Status> { + let cache_dir = image_cache_dir(&self.config.state_dir, image_identity); + let archive_path = image_cache_rootfs_archive(&self.config.state_dir, image_identity); + let staging_dir = image_cache_staging_dir(&self.config.state_dir, image_identity); + let exported_rootfs = staging_dir.join(IMAGE_EXPORT_ROOTFS_ARCHIVE); + let prepared_rootfs = staging_dir.join("rootfs"); + let prepared_archive = staging_dir.join(IMAGE_CACHE_ROOTFS_ARCHIVE); + + tokio::fs::create_dir_all(image_cache_root_dir(&self.config.state_dir)) + .await + .map_err(|err| Status::internal(format!("create image cache dir failed: {err}")))?; + tokio::fs::create_dir_all(&cache_dir) + .await + .map_err(|err| Status::internal(format!("create image cache dir failed: {err}")))?; + + if tokio::fs::metadata(&staging_dir).await.is_ok() { + tokio::fs::remove_dir_all(&staging_dir) + .await + .map_err(|err| { + Status::internal(format!( + "remove stale image cache staging dir failed: {err}" + )) + })?; + } + tokio::fs::create_dir_all(&staging_dir) + .await + .map_err(|err| { + Status::internal(format!("create image cache staging dir failed: {err}")) + })?; + + if let Err(err) = + export_local_image_rootfs_to_path(docker, image_ref, &exported_rootfs).await + { + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + return Err(err); + } + + let image_ref_owned = image_ref.to_string(); + let image_identity_owned = image_identity.to_string(); + let exported_rootfs_for_build = exported_rootfs.clone(); + let prepared_rootfs_for_build = prepared_rootfs.clone(); + let prepared_archive_for_build = prepared_archive.clone(); + let build_result = tokio::task::spawn_blocking(move || { + prepare_exported_rootfs_archive( + &image_ref_owned, + &image_identity_owned, + &exported_rootfs_for_build, + &prepared_rootfs_for_build, + &prepared_archive_for_build, + ) + }) + .await + .map_err(|err| Status::internal(format!("local image preparation panicked: {err}")))?; + + if let Err(err) = build_result { + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + return Err(Status::failed_precondition(err)); + } + + if tokio::fs::metadata(&archive_path).await.is_ok() { + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + return Ok(()); + } + + tokio::fs::rename(&prepared_archive, &archive_path) + .await + .map_err(|err| Status::internal(format!("store cached image rootfs failed: {err}")))?; + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + Ok(()) + } + + async fn build_cached_registry_image_rootfs_archive( + &self, + sandbox_id: &str, + client: &OciClient, + reference: &Reference, + auth: &RegistryAuth, + image_ref: &str, + image_identity: &str, + ) -> Result<(), Status> { + let cache_dir = image_cache_dir(&self.config.state_dir, image_identity); + let archive_path = image_cache_rootfs_archive(&self.config.state_dir, image_identity); + let staging_dir = image_cache_staging_dir(&self.config.state_dir, image_identity); + let prepared_rootfs = staging_dir.join("rootfs"); + let prepared_archive = staging_dir.join(IMAGE_CACHE_ROOTFS_ARCHIVE); + + tokio::fs::create_dir_all(image_cache_root_dir(&self.config.state_dir)) + .await + .map_err(|err| Status::internal(format!("create image cache dir failed: {err}")))?; + tokio::fs::create_dir_all(&cache_dir) + .await + .map_err(|err| Status::internal(format!("create image cache dir failed: {err}")))?; + + if tokio::fs::metadata(&staging_dir).await.is_ok() { + tokio::fs::remove_dir_all(&staging_dir) + .await + .map_err(|err| { + Status::internal(format!( + "remove stale image cache staging dir failed: {err}" + )) + })?; + } + tokio::fs::create_dir_all(&staging_dir) + .await + .map_err(|err| { + Status::internal(format!("create image cache staging dir failed: {err}")) + })?; + + info!( + image_ref = %image_ref, + staging_dir = %staging_dir.display(), + "vm driver: pulling registry image layers" + ); + if let Err(err) = self + .pull_registry_image_rootfs( + sandbox_id, + client, + reference, + auth, + image_ref, + &staging_dir, + &prepared_rootfs, + ) + .await + { + warn!( + image_ref = %image_ref, + error = %err.message(), + "vm driver: pull_registry_image_rootfs failed" + ); + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + return Err(err); + } + info!( + image_ref = %image_ref, + "vm driver: image layers pulled, preparing rootfs archive" + ); + + let image_ref_owned = image_ref.to_string(); + let image_identity_owned = image_identity.to_string(); + let prepared_rootfs_for_build = prepared_rootfs.clone(); + let prepared_archive_for_build = prepared_archive.clone(); + let build_result = tokio::task::spawn_blocking(move || { + prepare_sandbox_rootfs_from_image_root( + &prepared_rootfs_for_build, + &image_identity_owned, + ) + .map_err(|err| { + format!("vm sandbox image '{image_ref_owned}' is not base-compatible: {err}") + })?; + create_rootfs_archive_from_dir(&prepared_rootfs_for_build, &prepared_archive_for_build) + }) + .await + .map_err(|err| Status::internal(format!("image rootfs preparation panicked: {err}")))?; + + if let Err(err) = build_result { + warn!( + image_ref = %image_ref, + error = %err, + "vm driver: rootfs archive build failed" + ); + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + return Err(Status::failed_precondition(err)); + } + + if tokio::fs::metadata(&archive_path).await.is_ok() { + info!( + image_identity = %image_identity, + "vm driver: another task wrote archive while we were building, discarding ours" + ); + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + return Ok(()); + } + + tokio::fs::rename(&prepared_archive, &archive_path) + .await + .map_err(|err| Status::internal(format!("store cached image rootfs failed: {err}")))?; + info!( + image_identity = %image_identity, + archive_path = %archive_path.display(), + "vm driver: image rootfs archive committed to cache" + ); + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + Ok(()) + } + + /// Watch the launcher child process and surface errors as driver + /// conditions. + /// + /// The driver no longer owns the `Ready` transition — the gateway + /// promotes a sandbox to `Ready` the moment its supervisor session + /// lands (see `openshell-server/src/compute/mod.rs`). This loop only + /// handles the sad paths: the child process failing to start, exiting + /// abnormally, or becoming unpollable. Those still surface as driver + /// `Error` conditions so the gateway can reason about a dead VM. + async fn monitor_sandbox(&self, sandbox_id: String) { + loop { + let process = { + let registry = self.registry.lock().await; + let Some(record) = registry.get(&sandbox_id) else { + return; + }; + record.process.clone() + }; + + let exit_status = { + let mut process = process.lock().await; + if process.deleting { + return; + } + match process.child.try_wait() { + Ok(status) => status, + Err(err) => { + if let Some(snapshot) = self + .set_snapshot_condition( + &sandbox_id, + error_condition("ProcessPollFailed", &err.to_string()), + false, + ) + .await + { + self.publish_snapshot(snapshot); + } + self.publish_platform_event( + sandbox_id.clone(), + platform_event( + "vm", + "Warning", + "ProcessPollFailed", + format!("Failed to poll VM helper process: {err}"), + ), + ); + return; + } + } + }; + + if let Some(status) = exit_status { + let message = status.code().map_or_else( + || "VM process exited".to_string(), + |code| format!("VM process exited with status {code}"), + ); + if let Some(snapshot) = self + .set_snapshot_condition( + &sandbox_id, + error_condition("ProcessExited", &message), + false, + ) + .await + { + self.publish_snapshot(snapshot); + } + self.publish_platform_event( + sandbox_id.clone(), + platform_event("vm", "Warning", "ProcessExited", message), + ); + let has_gpu = { + let registry = self.registry.lock().await; + registry + .get(&sandbox_id) + .and_then(|r| r.gpu_bdf.as_ref()) + .is_some() + }; + if has_gpu { + self.release_gpu_and_subnet(&sandbox_id); + } + return; + } + + tokio::time::sleep(Duration::from_millis(250)).await; + } + } + + async fn set_snapshot_condition( + &self, + sandbox_id: &str, + condition: SandboxCondition, + deleting: bool, + ) -> Option { + let mut registry = self.registry.lock().await; + let record = registry.get_mut(sandbox_id)?; + record.snapshot.status = Some(status_with_condition(&record.snapshot, condition, deleting)); + Some(record.snapshot.clone()) + } + + fn publish_snapshot(&self, sandbox: Sandbox) { + let _ = self.events.send(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { + sandbox: Some(sandbox), + }, + )), + }); + } + + fn publish_deleted(&self, sandbox_id: String) { + let _ = self.events.send(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Deleted( + WatchSandboxesDeletedEvent { sandbox_id }, + )), + }); + } + + fn publish_platform_event(&self, sandbox_id: String, event: PlatformEvent) { + let _ = self.events.send(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::PlatformEvent( + WatchSandboxesPlatformEvent { + sandbox_id, + event: Some(event), + }, + )), + }); + } +} + +#[tonic::async_trait] +impl ComputeDriver for VmDriver { + async fn get_capabilities( + &self, + _request: Request, + ) -> Result, Status> { + Ok(Response::new(self.capabilities())) + } + + async fn validate_sandbox_create( + &self, + request: Request, + ) -> Result, Status> { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + self.validate_sandbox(&sandbox)?; + Ok(Response::new(ValidateSandboxCreateResponse {})) + } + + async fn create_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + let response = self.create_sandbox(&sandbox).await?; + Ok(Response::new(response)) + } + + async fn get_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + if request.sandbox_id.is_empty() && request.sandbox_name.is_empty() { + return Err(Status::invalid_argument( + "sandbox_id or sandbox_name is required", + )); + } + + let sandbox = self + .get_sandbox(&request.sandbox_id, &request.sandbox_name) + .await? + .ok_or_else(|| Status::not_found("sandbox not found"))?; + + if !request.sandbox_id.is_empty() && request.sandbox_id != sandbox.id { + return Err(Status::failed_precondition( + "sandbox_id did not match the fetched sandbox", + )); + } + + Ok(Response::new(GetSandboxResponse { + sandbox: Some(sandbox), + })) + } + + async fn list_sandboxes( + &self, + _request: Request, + ) -> Result, Status> { + Ok(Response::new(ListSandboxesResponse { + sandboxes: self.current_snapshots().await, + })) + } + + async fn stop_sandbox( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented( + "stop sandbox is not implemented by the vm compute driver", + )) + } + + async fn delete_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + let response = self + .delete_sandbox(&request.sandbox_id, &request.sandbox_name) + .await?; + Ok(Response::new(response)) + } + + type WatchSandboxesStream = + Pin> + Send + 'static>>; + + async fn watch_sandboxes( + &self, + _request: Request, + ) -> Result, Status> { + let initial = self.current_snapshots().await; + let mut rx = self.events.subscribe(); + let (tx, out_rx) = mpsc::channel(WATCH_BUFFER); + tokio::spawn(async move { + let mut sent = HashSet::new(); + for sandbox in initial { + sent.insert(sandbox.id.clone()); + if tx + .send(Ok(WatchSandboxesEvent { + payload: Some(watch_sandboxes_event::Payload::Sandbox( + WatchSandboxesSandboxEvent { + sandbox: Some(sandbox), + }, + )), + })) + .await + .is_err() + { + return; + } + } + + loop { + match rx.recv().await { + Ok(event) => { + if let Some(watch_sandboxes_event::Payload::Sandbox(sandbox_event)) = + &event.payload + && let Some(sandbox) = &sandbox_event.sandbox + && !sent.insert(sandbox.id.clone()) + { + // duplicate snapshots are still forwarded + } + if tx.send(Ok(event)).await.is_err() { + return; + } + } + Err(broadcast::error::RecvError::Lagged(_)) => {} + Err(broadcast::error::RecvError::Closed) => return, + } + } + }); + + Ok(Response::new(Box::pin(ReceiverStream::new(out_rx)))) + } +} + +#[cfg(target_os = "linux")] +#[allow(unsafe_code)] // libc::geteuid is a thin syscall wrapper +fn check_gpu_privileges() -> Result<(), String> { + if unsafe { libc::geteuid() } != 0 { + return Err( + "GPU support requires root privileges for VFIO bind/unbind and TAP networking. \ + Run with sudo or ensure CAP_SYS_ADMIN + CAP_NET_ADMIN capabilities are set." + .to_string(), + ); + } + Ok(()) +} + +// `tonic::Status` is ~176 bytes; it's the standard error type across the +// gRPC API surface, so boxing here would diverge from every other handler. +#[allow(clippy::result_large_err)] +fn validate_vm_sandbox(sandbox: &Sandbox, gpu_enabled: bool) -> Result<(), Status> { + let spec = sandbox + .spec + .as_ref() + .ok_or_else(|| Status::invalid_argument("sandbox spec is required"))?; + + if spec.gpu && !gpu_enabled { + return Err(Status::failed_precondition( + "GPU support is not enabled on this driver; start with --gpu", + )); + } + + if !spec.gpu && !spec.gpu_device.is_empty() { + return Err(Status::invalid_argument("gpu_device requires gpu=true")); + } + + if let Some(template) = spec.template.as_ref() { + if !template.agent_socket_path.is_empty() { + return Err(Status::failed_precondition( + "vm sandboxes do not support template.agent_socket_path", + )); + } + if template.platform_config.is_some() { + return Err(Status::failed_precondition( + "vm sandboxes do not support template.platform_config", + )); + } + if template.resources.is_some() { + return Err(Status::failed_precondition( + "vm sandboxes do not support template.resources", + )); + } + } + Ok(()) +} + +#[allow(clippy::result_large_err)] +fn parse_registry_reference(image_ref: &str) -> Result { + Reference::try_from(image_ref).map_err(|err| { + Status::failed_precondition(format!( + "invalid vm sandbox image reference '{image_ref}': {err}" + )) + }) +} + +fn is_openshell_local_build_image_ref(image_ref: &str) -> bool { + image_ref.starts_with("openshell/sandbox-from:") +} + +fn local_docker_image_platform_mismatch( + image_ref: &str, + actual_os: Option<&str>, + actual_arch: Option<&str>, +) -> Option { + let actual_os = actual_os.unwrap_or("unknown"); + let actual_arch = actual_arch.unwrap_or("unknown"); + let expected_os = "linux"; + let expected_arch = linux_oci_arch(); + + (actual_os != expected_os || actual_arch != expected_arch).then(|| { + format!( + "local Docker image '{image_ref}' is {actual_os}/{actual_arch}, but VM sandboxes require {expected_os}/{expected_arch}" + ) + }) +} + +fn is_docker_not_found_error(err: &BollardError) -> bool { + matches!( + err, + BollardError::DockerResponseServerError { + status_code: 404, + .. + } + ) +} + +async fn export_local_image_rootfs_to_path( + docker: &Docker, + image_ref: &str, + tar_path: &Path, +) -> Result<(), Status> { + let container_name = format!( + "openshell-vm-rootfs-export-{}", + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_nanos() + ); + let create_options = CreateContainerOptionsBuilder::default() + .name(container_name.as_str()) + .build(); + let container = docker + .create_container( + Some(create_options), + ContainerCreateBody { + image: Some(image_ref.to_string()), + ..Default::default() + }, + ) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to create temporary export container for local Docker image '{image_ref}': {err}" + )) + })?; + let container_id = container.id; + + let export_result = async { + if let Some(parent) = tar_path.parent() { + tokio::fs::create_dir_all(parent).await.map_err(|err| { + Status::internal(format!( + "create export dir {} failed: {err}", + parent.display() + )) + })?; + } + let mut file = tokio::fs::File::create(tar_path).await.map_err(|err| { + Status::internal(format!("create {} failed: {err}", tar_path.display())) + })?; + let mut stream = docker.export_container(&container_id); + while let Some(chunk) = stream.next().await { + let chunk = chunk.map_err(|err| { + Status::failed_precondition(format!( + "failed to export local Docker image '{image_ref}': {err}" + )) + })?; + file.write_all(&chunk).await.map_err(|err| { + Status::internal(format!("write {} failed: {err}", tar_path.display())) + })?; + } + file.flush() + .await + .map_err(|err| Status::internal(format!("flush {} failed: {err}", tar_path.display()))) + } + .await; + + let cleanup_result = docker + .remove_container( + &container_id, + Some(RemoveContainerOptionsBuilder::default().force(true).build()), + ) + .await; + + match (export_result, cleanup_result) { + (Ok(()), Ok(())) => Ok(()), + (Err(err), _) => Err(err), + (Ok(()), Err(err)) => Err(Status::internal(format!( + "failed to remove temporary export container for local Docker image '{image_ref}': {err}" + ))), + } +} + +fn prepare_exported_rootfs_archive( + image_ref: &str, + image_identity: &str, + exported_rootfs: &Path, + prepared_rootfs: &Path, + prepared_archive: &Path, +) -> Result<(), String> { + extract_rootfs_archive_to(exported_rootfs, prepared_rootfs)?; + prepare_sandbox_rootfs_from_image_root(prepared_rootfs, image_identity) + .map_err(|err| format!("vm sandbox image '{image_ref}' is not base-compatible: {err}"))?; + create_rootfs_archive_from_dir(prepared_rootfs, prepared_archive) +} + +fn registry_client() -> OciClient { + OciClient::new(ClientConfig { + platform_resolver: Some(Box::new(linux_platform_resolver)), + ..Default::default() + }) +} + +fn linux_platform_resolver(manifests: &[ImageIndexEntry]) -> Option { + let expected_arch = linux_oci_arch(); + manifests + .iter() + .find_map(|entry| { + let platform = entry.platform.as_ref()?; + (platform.os.to_string() == "linux" + && platform.architecture.to_string() == expected_arch) + .then(|| entry.digest.clone()) + }) + .or_else(|| { + manifests.iter().find_map(|entry| { + let platform = entry.platform.as_ref()?; + (platform.os.to_string() == "linux").then(|| entry.digest.clone()) + }) + }) +} + +fn linux_oci_arch() -> &'static str { + match std::env::consts::ARCH { + "x86_64" => "amd64", + "aarch64" => "arm64", + "arm" => "arm", + other => other, + } +} + +#[allow(clippy::result_large_err)] +fn registry_auth(image_ref: &str) -> Result { + let username = env_non_empty("OPENSHELL_REGISTRY_USERNAME"); + let token = env_non_empty("OPENSHELL_REGISTRY_TOKEN"); + + match token { + Some(token) => { + let username = match username { + Some(username) => username, + None if image_reference_registry_host(image_ref) + .eq_ignore_ascii_case("ghcr.io") => + { + "__token__".to_string() + } + None => { + return Err(Status::failed_precondition( + "OPENSHELL_REGISTRY_USERNAME is required when OPENSHELL_REGISTRY_TOKEN is set for non-GHCR registries", + )); + } + }; + Ok(RegistryAuth::Basic(username, token)) + } + None => Ok(RegistryAuth::Anonymous), + } +} + +fn env_non_empty(key: &str) -> Option { + std::env::var(key) + .ok() + .filter(|value| !value.trim().is_empty()) +} + +fn image_reference_registry_host(image_ref: &str) -> &str { + let mut parts = image_ref.splitn(2, '/'); + let first = parts.next().unwrap_or(image_ref); + let has_path = parts.next().is_some(); + if has_path + && (first.contains('.') || first.contains(':') || first.eq_ignore_ascii_case("localhost")) + { + first + } else { + "docker.io" + } +} + +impl VmDriver { + #[allow(clippy::too_many_arguments)] + async fn pull_registry_image_rootfs( + &self, + sandbox_id: &str, + client: &OciClient, + reference: &Reference, + auth: &RegistryAuth, + image_ref: &str, + staging_dir: &Path, + rootfs: &Path, + ) -> Result<(), Status> { + client + .auth(reference, auth, RegistryOperation::Pull) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to authenticate registry access for vm sandbox image '{image_ref}': {err}" + )) + })?; + let (manifest, _) = client + .pull_image_manifest(reference, auth) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to pull vm sandbox image manifest '{image_ref}': {err}" + )) + })?; + + tokio::fs::create_dir_all(rootfs) + .await + .map_err(|err| Status::internal(format!("create rootfs dir failed: {err}")))?; + tokio::fs::create_dir_all(staging_dir.join("layers")) + .await + .map_err(|err| Status::internal(format!("create layer staging dir failed: {err}")))?; + + let total_layers = manifest.layers.len(); + let total_bytes: i64 = manifest.layers.iter().map(|layer| layer.size.max(0)).sum(); + for (index, layer) in manifest.layers.iter().enumerate() { + // Emit a per-layer progress event so the CLI can show + // "Layer 3/8 (12.4 MB)" as detail under the spinner. + let mut metadata = HashMap::new(); + metadata.insert("layer_index".to_string(), (index + 1).to_string()); + metadata.insert("layer_total".to_string(), total_layers.to_string()); + metadata.insert("layer_digest".to_string(), layer.digest.clone()); + metadata.insert("layer_size_bytes".to_string(), layer.size.to_string()); + metadata.insert("image_ref".to_string(), image_ref.to_string()); + if total_bytes > 0 { + metadata.insert("image_size_bytes".to_string(), total_bytes.to_string()); + } + let mut event = platform_event( + "vm", + "Normal", + "PullingLayer", + format!( + "Pulling layer {}/{} ({} bytes) for image \"{image_ref}\"", + index + 1, + total_layers, + layer.size + ), + ); + event.metadata = metadata; + self.publish_platform_event(sandbox_id.to_string(), event); + + pull_registry_layer( + client, + reference, + image_ref, + staging_dir, + rootfs, + layer, + index, + ) + .await?; + } + + Ok(()) + } + + /// Emit a `Pulled` platform event with a message that mirrors the + /// kubelet's `Successfully pulled image ... Image size: N bytes.` + /// format so the CLI's `extract_image_size` parser works unchanged. + async fn publish_pulled_event(&self, sandbox_id: &str, image_ref: &str, archive_path: &Path) { + let size_suffix = tokio::fs::metadata(archive_path).await.map_or_else( + |_| String::new(), + |meta| format!(" Image size: {} bytes.", meta.len()), + ); + self.publish_platform_event( + sandbox_id.to_string(), + platform_event( + "vm", + "Normal", + "Pulled", + format!("Successfully pulled image \"{image_ref}\".{size_suffix}"), + ), + ); + } +} + +async fn pull_registry_layer( + client: &OciClient, + reference: &Reference, + image_ref: &str, + staging_dir: &Path, + rootfs: &Path, + layer: &OciDescriptor, + index: usize, +) -> Result<(), Status> { + let digest_component = sanitize_image_identity(&layer.digest); + let blob_path = staging_dir + .join("layers") + .join(format!("{index:02}-{digest_component}.blob")); + let layer_root = staging_dir + .join("layers") + .join(format!("{index:02}-{digest_component}.root")); + + let mut file = tokio::fs::File::create(&blob_path) + .await + .map_err(|err| Status::internal(format!("create layer blob failed: {err}")))?; + client + .pull_blob(reference, layer, &mut file) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to download layer '{}' for vm sandbox image '{image_ref}': {err}", + layer.digest + )) + })?; + file.flush() + .await + .map_err(|err| Status::internal(format!("flush layer blob failed: {err}")))?; + + let blob_path_for_digest = blob_path.clone(); + let expected_digest = layer.digest.clone(); + tokio::task::spawn_blocking(move || { + verify_descriptor_digest(&blob_path_for_digest, &expected_digest) + }) + .await + .map_err(|err| Status::internal(format!("layer digest verification panicked: {err}")))? + .map_err(|err| { + Status::failed_precondition(format!( + "vm sandbox image layer verification failed for '{}': {err}", + layer.digest + )) + })?; + + let blob_path_for_unpack = blob_path.clone(); + let layer_root_for_unpack = layer_root.clone(); + let rootfs_for_unpack = rootfs.to_path_buf(); + let media_type = layer.media_type.clone(); + tokio::task::spawn_blocking(move || { + extract_layer_blob_to_dir(&blob_path_for_unpack, &media_type, &layer_root_for_unpack)?; + apply_layer_dir_to_rootfs(&layer_root_for_unpack, &rootfs_for_unpack) + }) + .await + .map_err(|err| Status::internal(format!("layer extraction panicked: {err}")))? + .map_err(|err| { + Status::failed_precondition(format!( + "failed to apply layer '{}' for vm sandbox image '{image_ref}': {err}", + layer.digest + )) + }) +} + +fn verify_descriptor_digest(path: &Path, expected_digest: &str) -> Result<(), String> { + let expected = expected_digest + .strip_prefix("sha256:") + .ok_or_else(|| format!("unsupported layer digest '{expected_digest}'"))?; + let actual = compute_file_sha256_hex(path)?; + if actual == expected { + Ok(()) + } else { + Err(format!( + "digest mismatch for {}: expected sha256:{expected}, got sha256:{actual}", + path.display() + )) + } +} + +fn compute_file_sha256_hex(path: &Path) -> Result { + let mut file = fs::File::open(path).map_err(|err| format!("open {}: {err}", path.display()))?; + let mut hasher = Sha256::new(); + let mut buffer = vec![0_u8; 64 * 1024].into_boxed_slice(); + loop { + let read = file + .read(&mut buffer) + .map_err(|err| format!("read {}: {err}", path.display()))?; + if read == 0 { + break; + } + hasher.update(&buffer[..read]); + } + Ok(format!("{:x}", hasher.finalize())) +} + +fn extract_layer_blob_to_dir( + blob_path: &Path, + media_type: &str, + dest: &Path, +) -> Result<(), String> { + if dest.exists() { + fs::remove_dir_all(dest).map_err(|err| format!("remove {}: {err}", dest.display()))?; + } + fs::create_dir_all(dest).map_err(|err| format!("create {}: {err}", dest.display()))?; + + let file = + fs::File::open(blob_path).map_err(|err| format!("open {}: {err}", blob_path.display()))?; + match layer_compression_from_media_type(media_type)? { + LayerCompression::None => extract_tar_reader_to_dir(file, dest), + LayerCompression::Gzip => extract_tar_reader_to_dir(GzDecoder::new(file), dest), + LayerCompression::Zstd => { + let decoder = zstd::stream::read::Decoder::new(file) + .map_err(|err| format!("decompress {}: {err}", blob_path.display()))?; + extract_tar_reader_to_dir(decoder, dest) + } + } +} + +fn extract_tar_reader_to_dir(reader: impl Read, dest: &Path) -> Result<(), String> { + let mut archive = tar::Archive::new(reader); + archive + .unpack(dest) + .map_err(|err| format!("extract layer into {}: {err}", dest.display())) +} + +// `media_type` is an OCI media type string (e.g. `application/vnd.oci.image.layer.v1.tar+gzip`), +// not a filesystem path, so case-sensitive comparison is correct. +#[allow(clippy::case_sensitive_file_extension_comparisons)] +fn layer_compression_from_media_type(media_type: &str) -> Result { + if media_type.is_empty() { + return Err("layer media type is missing".to_string()); + } + if media_type.ends_with("+zstd") { + return Ok(LayerCompression::Zstd); + } + if media_type.ends_with("+gzip") || media_type.ends_with(".gzip") { + return Ok(LayerCompression::Gzip); + } + if media_type.ends_with(".tar") + || media_type.ends_with("tar") + || media_type == "application/vnd.oci.image.layer.v1.tar" + || media_type == "application/vnd.oci.image.layer.nondistributable.v1.tar" + { + return Ok(LayerCompression::None); + } + Err(format!("unsupported layer media type '{media_type}'")) +} + +fn apply_layer_dir_to_rootfs(layer_root: &Path, rootfs: &Path) -> Result<(), String> { + merge_layer_directory(layer_root, rootfs) +} + +fn merge_layer_directory(source_dir: &Path, target_dir: &Path) -> Result<(), String> { + fs::create_dir_all(target_dir) + .map_err(|err| format!("create {}: {err}", target_dir.display()))?; + + let mut entries = fs::read_dir(source_dir) + .map_err(|err| format!("read {}: {err}", source_dir.display()))? + .collect::, _>>() + .map_err(|err| format!("read {}: {err}", source_dir.display()))?; + entries.sort_by_key(fs::DirEntry::file_name); + + if entries + .iter() + .any(|entry| entry.file_name().to_string_lossy() == ".wh..wh..opq") + { + clear_directory_contents(target_dir)?; + } + + for entry in entries { + let file_name = entry.file_name(); + let name = file_name.to_string_lossy(); + if name == ".wh..wh..opq" { + continue; + } + if let Some(hidden_name) = name.strip_prefix(".wh.") { + remove_path_if_exists(&target_dir.join(hidden_name))?; + continue; + } + + let source_path = entry.path(); + let dest_path = target_dir.join(&file_name); + let metadata = fs::symlink_metadata(&source_path) + .map_err(|err| format!("stat {}: {err}", source_path.display()))?; + let file_type = metadata.file_type(); + + if file_type.is_dir() { + if let Ok(dest_metadata) = fs::symlink_metadata(&dest_path) + && !dest_metadata.file_type().is_dir() + && !path_is_dir_or_symlink_to_dir(&dest_path)? + { + remove_path_if_exists(&dest_path)?; + } + fs::create_dir_all(&dest_path) + .map_err(|err| format!("create {}: {err}", dest_path.display()))?; + merge_layer_directory(&source_path, &dest_path)?; + if fs::symlink_metadata(&dest_path) + .map_err(|err| format!("stat {}: {err}", dest_path.display()))? + .file_type() + .is_dir() + { + fs::set_permissions(&dest_path, metadata.permissions()) + .map_err(|err| format!("chmod {}: {err}", dest_path.display()))?; + } + } else if file_type.is_file() { + remove_path_if_exists(&dest_path)?; + if let Some(parent) = dest_path.parent() { + fs::create_dir_all(parent) + .map_err(|err| format!("create {}: {err}", parent.display()))?; + } + fs::copy(&source_path, &dest_path).map_err(|err| { + format!( + "copy {} to {}: {err}", + source_path.display(), + dest_path.display() + ) + })?; + fs::set_permissions(&dest_path, metadata.permissions()) + .map_err(|err| format!("chmod {}: {err}", dest_path.display()))?; + } else if file_type.is_symlink() { + copy_symlink(&source_path, &dest_path)?; + } else { + return Err(format!( + "unsupported layer entry type at {}", + source_path.display() + )); + } + } + + Ok(()) +} + +fn path_is_dir_or_symlink_to_dir(path: &Path) -> Result { + match fs::metadata(path) { + Ok(metadata) => Ok(metadata.file_type().is_dir()), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(false), + Err(err) => Err(format!("stat {}: {err}", path.display())), + } +} + +fn clear_directory_contents(dir: &Path) -> Result<(), String> { + if !dir.exists() { + return Ok(()); + } + for entry in fs::read_dir(dir).map_err(|err| format!("read {}: {err}", dir.display()))? { + let entry = entry.map_err(|err| format!("read {}: {err}", dir.display()))?; + remove_path_if_exists(&entry.path())?; + } + Ok(()) +} + +fn remove_path_if_exists(path: &Path) -> Result<(), String> { + let Ok(metadata) = fs::symlink_metadata(path) else { + return Ok(()); + }; + if metadata.file_type().is_dir() { + fs::remove_dir_all(path).map_err(|err| format!("remove {}: {err}", path.display())) + } else { + fs::remove_file(path).map_err(|err| format!("remove {}: {err}", path.display())) + } +} + +#[cfg(unix)] +fn copy_symlink(source_path: &Path, dest_path: &Path) -> Result<(), String> { + let target = fs::read_link(source_path) + .map_err(|err| format!("readlink {}: {err}", source_path.display()))?; + remove_path_if_exists(dest_path)?; + if let Some(parent) = dest_path.parent() { + fs::create_dir_all(parent).map_err(|err| format!("create {}: {err}", parent.display()))?; + } + std::os::unix::fs::symlink(&target, dest_path).map_err(|err| { + format!( + "symlink {} to {}: {err}", + target.display(), + dest_path.display() + ) + }) +} + +#[cfg(not(unix))] +fn copy_symlink(_source_path: &Path, _dest_path: &Path) -> Result<(), String> { + Err("symlink layers are only supported on Unix hosts".to_string()) +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum LayerCompression { + None, + Gzip, + Zstd, +} + +fn requested_sandbox_image(sandbox: &Sandbox) -> Option<&str> { + sandbox + .spec + .as_ref() + .and_then(|spec| spec.template.as_ref()) + .map(|template| template.image.trim()) + .filter(|image| !image.is_empty()) +} + +fn merged_environment(sandbox: &Sandbox) -> HashMap { + let mut environment = sandbox + .spec + .as_ref() + .and_then(|spec| spec.template.as_ref()) + .map_or_else(HashMap::new, |template| template.environment.clone()); + if let Some(spec) = sandbox.spec.as_ref() { + environment.extend(spec.environment.clone()); + } + environment +} + +/// Rewrites loopback host references in a gateway URL to a hostname the guest +/// can reach via gvproxy. +/// +/// The driver receives the gateway endpoint from `--openshell-endpoint`, which +/// in local/dev/e2e setups is typically `http://127.0.0.1:`. That URL is +/// useless inside the guest because the guest's loopback interface is its own, +/// not the host's. Inside the guest we need a name that gvproxy will translate +/// into the host's loopback address. +/// +/// We rewrite to `host.containers.internal`, which gvproxy's embedded DNS resolves +/// to the host-loopback IP `192.168.127.254`. gvproxy installs a default NAT entry +/// rewriting that destination to the host's `127.0.0.1` and dialing out from the +/// host process, so any port the host is listening on becomes reachable. The +/// gateway IP `192.168.127.1` does **not** do this — it only listens on gvproxy's +/// own service ports (DNS, DHCP, HTTP API). The guest init script also seeds the +/// hostname in `/etc/hosts` so resolution works even if gvproxy's DNS isn't in +/// resolv.conf (e.g. when DHCP fails). +/// +/// Non-loopback URLs are returned unchanged. +fn guest_visible_openshell_endpoint(endpoint: &str) -> String { + let Ok(mut url) = Url::parse(endpoint) else { + return endpoint.to_string(); + }; + + let should_rewrite = match url.host() { + Some(Host::Ipv4(ip)) => ip.is_loopback(), + Some(Host::Ipv6(ip)) => ip.is_loopback(), + Some(Host::Domain(host)) => host.eq_ignore_ascii_case("localhost"), + None => false, + }; + + if should_rewrite && url.set_host(Some(GVPROXY_HOST_LOOPBACK_ALIAS)).is_ok() { + return url.to_string(); + } + + endpoint.to_string() +} + +fn gateway_port_from_endpoint(endpoint: &str) -> Option { + Url::parse(endpoint).ok().and_then(|url| url.port()) +} + +fn guest_visible_openshell_endpoint_for_tap(endpoint: &str, host_ip: &str) -> String { + let Ok(mut url) = Url::parse(endpoint) else { + return endpoint.to_string(); + }; + if url.set_host(Some(host_ip)).is_ok() { + url.to_string() + } else { + endpoint.to_string() + } +} + +fn build_guest_environment( + sandbox: &Sandbox, + config: &VmDriverConfig, + endpoint_override: Option<&str>, +) -> Vec { + let openshell_endpoint = endpoint_override.map_or_else( + || guest_visible_openshell_endpoint(&config.openshell_endpoint), + String::from, + ); + let mut environment = HashMap::from([ + ("HOME".to_string(), "/root".to_string()), + ( + "PATH".to_string(), + "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin".to_string(), + ), + ("TERM".to_string(), "xterm".to_string()), + ("OPENSHELL_ENDPOINT".to_string(), openshell_endpoint), + ("OPENSHELL_SANDBOX_ID".to_string(), sandbox.id.clone()), + ("OPENSHELL_SANDBOX".to_string(), sandbox.name.clone()), + ( + "OPENSHELL_SSH_SOCKET_PATH".to_string(), + GUEST_SSH_SOCKET_PATH.to_string(), + ), + ( + "OPENSHELL_SANDBOX_COMMAND".to_string(), + "tail -f /dev/null".to_string(), + ), + ( + "OPENSHELL_LOG_LEVEL".to_string(), + sandbox_log_level(sandbox, &config.log_level), + ), + ( + "OPENSHELL_SSH_HANDSHAKE_SECRET".to_string(), + config.ssh_handshake_secret.clone(), + ), + ]); + if config.requires_tls_materials() { + environment.extend(HashMap::from([ + ( + "OPENSHELL_TLS_CA".to_string(), + GUEST_TLS_CA_PATH.to_string(), + ), + ( + "OPENSHELL_TLS_CERT".to_string(), + GUEST_TLS_CERT_PATH.to_string(), + ), + ( + "OPENSHELL_TLS_KEY".to_string(), + GUEST_TLS_KEY_PATH.to_string(), + ), + ])); + } + environment.extend(merged_environment(sandbox)); + + let mut pairs = environment.into_iter().collect::>(); + pairs.sort_by(|left, right| left.0.cmp(&right.0)); + pairs + .into_iter() + .map(|(key, value)| format!("{key}={value}")) + .collect() +} + +fn sandbox_log_level(sandbox: &Sandbox, default_level: &str) -> String { + sandbox + .spec + .as_ref() + .map(|spec| spec.log_level.as_str()) + .filter(|level| !level.is_empty()) + .unwrap_or(default_level) + .to_string() +} + +fn sandboxes_root_dir(root: &Path) -> PathBuf { + root.join("sandboxes") +} + +fn sandbox_state_dir(root: &Path, sandbox_id: &str) -> PathBuf { + sandboxes_root_dir(root).join(sandbox_id) +} + +fn image_cache_root_dir(root: &Path) -> PathBuf { + root.join(IMAGE_CACHE_ROOT_DIR) +} + +fn image_cache_dir(root: &Path, image_identity: &str) -> PathBuf { + image_cache_root_dir(root).join(sanitize_image_identity(image_identity)) +} + +fn image_cache_rootfs_archive(root: &Path, image_identity: &str) -> PathBuf { + image_cache_dir(root, image_identity).join(IMAGE_CACHE_ROOTFS_ARCHIVE) +} + +fn image_cache_staging_dir(root: &Path, image_identity: &str) -> PathBuf { + image_cache_root_dir(root).join(format!( + "{}.staging-{}", + sanitize_image_identity(image_identity), + unique_image_cache_suffix() + )) +} + +fn sanitize_image_identity(image_identity: &str) -> String { + image_identity + .chars() + .map(|ch| { + if ch.is_ascii_alphanumeric() || ch == '-' || ch == '_' || ch == '.' { + ch + } else { + '-' + } + }) + .collect() +} + +fn unique_image_cache_suffix() -> String { + let counter = IMAGE_CACHE_BUILD_COUNTER.fetch_add(1, Ordering::Relaxed); + format!("{}-{counter}", current_time_ms()) +} + +async fn write_sandbox_image_metadata( + state_dir: &Path, + image_ref: &str, + image_identity: &str, +) -> Result<(), std::io::Error> { + tokio::fs::write( + state_dir.join(IMAGE_IDENTITY_FILE), + format!("{image_identity}\n"), + ) + .await?; + tokio::fs::write( + state_dir.join(IMAGE_REFERENCE_FILE), + format!("{image_ref}\n"), + ) + .await?; + + Ok(()) +} + +async fn prepare_guest_tls_materials( + rootfs: &Path, + paths: &VmDriverTlsPaths, +) -> Result<(), std::io::Error> { + let guest_tls_dir = rootfs.join(GUEST_TLS_DIR.trim_start_matches('/')); + tokio::fs::create_dir_all(&guest_tls_dir).await?; + + copy_guest_tls_material(&paths.ca, &guest_tls_dir.join("ca.crt"), 0o644).await?; + copy_guest_tls_material(&paths.cert, &guest_tls_dir.join("tls.crt"), 0o644).await?; + copy_guest_tls_material(&paths.key, &guest_tls_dir.join("tls.key"), 0o600).await?; + Ok(()) +} + +async fn copy_guest_tls_material( + source: &Path, + dest: &Path, + mode: u32, +) -> Result<(), std::io::Error> { + tokio::fs::copy(source, dest).await?; + tokio::fs::set_permissions(dest, fs::Permissions::from_mode(mode)).await?; + Ok(()) +} + +async fn terminate_vm_process(child: &mut Child) -> Result<(), std::io::Error> { + if let Some(pid) = child.id() + && let Err(err) = kill(Pid::from_raw(pid.cast_signed()), Signal::SIGTERM) + && err != Errno::ESRCH + { + return Err(std::io::Error::other(format!( + "send SIGTERM to vm process {pid}: {err}" + ))); + } + + match tokio::time::timeout(Duration::from_secs(5), child.wait()).await { + Ok(Ok(_)) => Ok(()), + Ok(Err(err)) => Err(err), + Err(_) => { + child.kill().await?; + child.wait().await.map(|_| ()) + } + } +} + +fn sandbox_snapshot(sandbox: &Sandbox, condition: SandboxCondition, deleting: bool) -> Sandbox { + Sandbox { + id: sandbox.id.clone(), + name: sandbox.name.clone(), + namespace: sandbox.namespace.clone(), + status: Some(SandboxStatus { + sandbox_name: sandbox.name.clone(), + instance_id: String::new(), + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: vec![condition], + deleting, + }), + ..Default::default() + } +} + +fn status_with_condition( + snapshot: &Sandbox, + condition: SandboxCondition, + deleting: bool, +) -> SandboxStatus { + SandboxStatus { + sandbox_name: snapshot.name.clone(), + instance_id: String::new(), + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: vec![condition], + deleting, + } +} + +fn provisioning_condition() -> SandboxCondition { + SandboxCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: "Starting".to_string(), + message: "VM is starting".to_string(), + last_transition_time: String::new(), + } +} + +fn deleting_condition() -> SandboxCondition { + SandboxCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: "Deleting".to_string(), + message: "Sandbox is being deleted".to_string(), + last_transition_time: String::new(), + } +} + +fn error_condition(reason: &str, message: &str) -> SandboxCondition { + SandboxCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: reason.to_string(), + message: message.to_string(), + last_transition_time: String::new(), + } +} + +fn platform_event(source: &str, event_type: &str, reason: &str, message: String) -> PlatformEvent { + PlatformEvent { + timestamp_ms: current_time_ms(), + source: source.to_string(), + r#type: event_type.to_string(), + reason: reason.to_string(), + message, + metadata: HashMap::new(), + } +} + +fn current_time_ms() -> i64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |duration| { + i64::try_from(duration.as_millis()).unwrap_or(i64::MAX) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::gpu::{SubnetAllocator, allocate_vsock_cid, mac_from_sandbox_id, tap_device_name}; + use openshell_core::proto::compute::v1::{ + DriverSandboxSpec as SandboxSpec, DriverSandboxTemplate as SandboxTemplate, + }; + use prost_types::{Struct, Value, value::Kind}; + use std::fs; + use std::sync::atomic::{AtomicU64, Ordering}; + use std::time::{SystemTime, UNIX_EPOCH}; + use tonic::Code; + + #[test] + fn validate_vm_sandbox_rejects_gpu_when_not_enabled() { + let sandbox = Sandbox { + spec: Some(SandboxSpec { + gpu: true, + ..Default::default() + }), + ..Default::default() + }; + let err = validate_vm_sandbox(&sandbox, false) + .expect_err("gpu should be rejected when not enabled"); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains("GPU support is not enabled")); + } + + #[test] + fn validate_vm_sandbox_accepts_gpu_when_enabled() { + let sandbox = Sandbox { + spec: Some(SandboxSpec { + gpu: true, + ..Default::default() + }), + ..Default::default() + }; + validate_vm_sandbox(&sandbox, true).expect("gpu should be accepted when enabled"); + } + + #[test] + fn validate_vm_sandbox_rejects_gpu_device_without_gpu() { + let sandbox = Sandbox { + spec: Some(SandboxSpec { + gpu: false, + gpu_device: "0000:2d:00.0".to_string(), + ..Default::default() + }), + ..Default::default() + }; + let err = validate_vm_sandbox(&sandbox, true) + .expect_err("gpu_device without gpu should be rejected"); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("gpu_device requires gpu=true")); + } + + #[test] + fn validate_vm_sandbox_rejects_platform_config() { + let sandbox = Sandbox { + spec: Some(SandboxSpec { + template: Some(SandboxTemplate { + platform_config: Some(Struct { + fields: std::iter::once(( + "runtime_class_name".to_string(), + Value { + kind: Some(Kind::StringValue("kata".to_string())), + }, + )) + .collect(), + }), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }; + let err = + validate_vm_sandbox(&sandbox, false).expect_err("platform config should be rejected"); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains("platform_config")); + } + + #[test] + fn validate_vm_sandbox_accepts_template_image() { + let sandbox = Sandbox { + spec: Some(SandboxSpec { + template: Some(SandboxTemplate { + image: "ghcr.io/example/sandbox:latest".to_string(), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }; + validate_vm_sandbox(&sandbox, false).expect("template.image should be accepted"); + } + + #[test] + fn capabilities_report_configured_default_image() { + let driver = VmDriver { + config: VmDriverConfig { + default_image: "openshell/sandbox:dev".to_string(), + ..Default::default() + }, + launcher_bin: PathBuf::from("/tmp/openshell-driver-vm"), + registry: Arc::new(Mutex::new(HashMap::new())), + image_cache_lock: Arc::new(Mutex::new(())), + events: broadcast::channel(WATCH_BUFFER).0, + gpu_inventory: None, + subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( + Ipv4Addr::new(10, 0, 128, 0), + 17, + ))), + }; + + assert_eq!(driver.capabilities().default_image, "openshell/sandbox:dev"); + } + + #[test] + fn resolved_sandbox_image_prefers_template_image() { + let driver = VmDriver { + config: VmDriverConfig { + default_image: "openshell/sandbox:default".to_string(), + ..Default::default() + }, + launcher_bin: PathBuf::from("/tmp/openshell-driver-vm"), + registry: Arc::new(Mutex::new(HashMap::new())), + image_cache_lock: Arc::new(Mutex::new(())), + events: broadcast::channel(WATCH_BUFFER).0, + gpu_inventory: None, + subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( + Ipv4Addr::new(10, 0, 128, 0), + 17, + ))), + }; + let sandbox = Sandbox { + spec: Some(SandboxSpec { + template: Some(SandboxTemplate { + image: "ghcr.io/example/custom:latest".to_string(), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }; + + assert_eq!( + driver.resolved_sandbox_image(&sandbox).as_deref(), + Some("ghcr.io/example/custom:latest") + ); + } + + #[test] + fn resolved_sandbox_image_falls_back_to_driver_default() { + let driver = VmDriver { + config: VmDriverConfig { + default_image: "openshell/sandbox:default".to_string(), + ..Default::default() + }, + launcher_bin: PathBuf::from("/tmp/openshell-driver-vm"), + registry: Arc::new(Mutex::new(HashMap::new())), + image_cache_lock: Arc::new(Mutex::new(())), + events: broadcast::channel(WATCH_BUFFER).0, + gpu_inventory: None, + subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( + Ipv4Addr::new(10, 0, 128, 0), + 17, + ))), + }; + let sandbox = Sandbox { + spec: Some(SandboxSpec { + template: Some(SandboxTemplate::default()), + ..Default::default() + }), + ..Default::default() + }; + + assert_eq!( + driver.resolved_sandbox_image(&sandbox).as_deref(), + Some("openshell/sandbox:default") + ); + } + + #[test] + fn resolved_sandbox_image_returns_none_without_template_or_default() { + let driver = VmDriver { + config: VmDriverConfig::default(), + launcher_bin: PathBuf::from("/tmp/openshell-driver-vm"), + registry: Arc::new(Mutex::new(HashMap::new())), + image_cache_lock: Arc::new(Mutex::new(())), + events: broadcast::channel(WATCH_BUFFER).0, + gpu_inventory: None, + subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( + Ipv4Addr::new(10, 0, 128, 0), + 17, + ))), + }; + let sandbox = Sandbox { + spec: Some(SandboxSpec { + template: Some(SandboxTemplate::default()), + ..Default::default() + }), + ..Default::default() + }; + + assert!(driver.resolved_sandbox_image(&sandbox).is_none()); + } + + #[test] + fn merged_environment_prefers_spec_values() { + let sandbox = Sandbox { + spec: Some(SandboxSpec { + environment: HashMap::from([("A".to_string(), "spec".to_string())]), + template: Some(SandboxTemplate { + environment: HashMap::from([ + ("A".to_string(), "template".to_string()), + ("B".to_string(), "template".to_string()), + ]), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }; + let merged = merged_environment(&sandbox); + assert_eq!(merged.get("A"), Some(&"spec".to_string())); + assert_eq!(merged.get("B"), Some(&"template".to_string())); + } + + #[test] + fn build_guest_environment_sets_supervisor_defaults() { + let config = VmDriverConfig { + openshell_endpoint: "http://127.0.0.1:8080".to_string(), + ssh_handshake_secret: "secret".to_string(), + ..Default::default() + }; + let sandbox = Sandbox { + id: "sandbox-123".to_string(), + name: "sandbox-123".to_string(), + spec: Some(SandboxSpec::default()), + ..Default::default() + }; + + let env = build_guest_environment(&sandbox, &config, None); + assert!(env.contains(&"HOME=/root".to_string())); + assert!(env.contains(&format!( + "OPENSHELL_ENDPOINT=http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080/" + ))); + assert!(env.contains(&"OPENSHELL_SANDBOX_ID=sandbox-123".to_string())); + assert!(env.contains(&format!( + "OPENSHELL_SSH_SOCKET_PATH={GUEST_SSH_SOCKET_PATH}" + ))); + assert!( + env.contains(&"OPENSHELL_SSH_HANDSHAKE_SECRET=secret".to_string()), + "SSH handshake secret must be passed to the guest" + ); + } + + #[test] + fn build_guest_environment_uses_endpoint_override_for_tap() { + let config = VmDriverConfig { + openshell_endpoint: "http://127.0.0.1:8080".to_string(), + ssh_handshake_secret: "secret".to_string(), + ..Default::default() + }; + let sandbox = Sandbox { + id: "sandbox-123".to_string(), + name: "sandbox-123".to_string(), + spec: Some(SandboxSpec::default()), + ..Default::default() + }; + + let env = build_guest_environment(&sandbox, &config, Some("http://10.0.128.1:8080")); + assert!( + env.contains(&"OPENSHELL_ENDPOINT=http://10.0.128.1:8080".to_string()), + "TAP endpoint override must replace the default" + ); + let endpoint_count = env + .iter() + .filter(|e| e.starts_with("OPENSHELL_ENDPOINT=")) + .count(); + assert_eq!( + endpoint_count, 1, + "must have exactly one OPENSHELL_ENDPOINT" + ); + } + + #[test] + fn guest_visible_openshell_endpoint_rewrites_loopback_hosts_to_gvproxy_host_alias() { + assert_eq!( + guest_visible_openshell_endpoint("http://127.0.0.1:8080"), + format!("http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080/") + ); + assert_eq!( + guest_visible_openshell_endpoint("http://localhost:8080"), + format!("http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080/") + ); + assert_eq!( + guest_visible_openshell_endpoint("https://[::1]:8443"), + format!("https://{GVPROXY_HOST_LOOPBACK_ALIAS}:8443/") + ); + } + + #[test] + fn guest_visible_openshell_endpoint_preserves_non_loopback_hosts() { + assert_eq!( + guest_visible_openshell_endpoint(&format!( + "http://{OPENSHELL_HOST_GATEWAY_ALIAS}:8080" + )), + format!("http://{OPENSHELL_HOST_GATEWAY_ALIAS}:8080") + ); + assert_eq!( + guest_visible_openshell_endpoint(&format!("http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080")), + format!("http://{GVPROXY_HOST_LOOPBACK_ALIAS}:8080") + ); + assert_eq!( + guest_visible_openshell_endpoint("http://192.168.127.1:8080"), + "http://192.168.127.1:8080" + ); + assert_eq!( + guest_visible_openshell_endpoint("https://gateway.internal:8443"), + "https://gateway.internal:8443" + ); + } + + #[test] + fn image_reference_registry_host_defaults_to_docker_hub() { + assert_eq!(image_reference_registry_host("ubuntu:24.04"), "docker.io"); + assert_eq!( + image_reference_registry_host("library/ubuntu:24.04"), + "docker.io" + ); + assert_eq!( + image_reference_registry_host("ghcr.io/nvidia/openshell/base:latest"), + "ghcr.io" + ); + assert_eq!( + image_reference_registry_host("localhost/example:dev"), + "localhost" + ); + assert_eq!( + image_reference_registry_host("localhost:5000/example/sandbox:dev"), + "localhost:5000" + ); + } + + #[test] + fn openshell_local_build_image_ref_matches_cli_tags() { + assert!(is_openshell_local_build_image_ref( + "openshell/sandbox-from:123" + )); + assert!(!is_openshell_local_build_image_ref("ubuntu:24.04")); + assert!(!is_openshell_local_build_image_ref( + "ghcr.io/nvidia/openshell/base:latest" + )); + } + + #[test] + fn local_docker_image_platform_mismatch_checks_guest_platform() { + assert!( + local_docker_image_platform_mismatch( + "openshell/sandbox-from:123", + Some("linux"), + Some(linux_oci_arch()), + ) + .is_none() + ); + + let err = local_docker_image_platform_mismatch( + "openshell/sandbox-from:123", + Some("linux"), + Some("wrong-arch"), + ) + .expect("architecture mismatch should be reported"); + assert!(err.contains("wrong-arch")); + assert!(err.contains(linux_oci_arch())); + + let err = local_docker_image_platform_mismatch("openshell/sandbox-from:123", None, None) + .expect("unknown platform should be reported"); + assert!(err.contains("unknown/unknown")); + } + + #[test] + fn apply_layer_dir_to_rootfs_honors_whiteouts() { + let base = unique_temp_dir(); + let rootfs = base.join("rootfs"); + let layer = base.join("layer"); + + fs::create_dir_all(rootfs.join("dir")).unwrap(); + fs::write(rootfs.join("removed.txt"), "old").unwrap(); + fs::write(rootfs.join("dir/old.txt"), "old").unwrap(); + + fs::create_dir_all(layer.join("dir")).unwrap(); + fs::write(layer.join(".wh.removed.txt"), "").unwrap(); + fs::write(layer.join("dir/.wh..wh..opq"), "").unwrap(); + fs::write(layer.join("dir/new.txt"), "new").unwrap(); + + apply_layer_dir_to_rootfs(&layer, &rootfs).unwrap(); + + assert!(!rootfs.join("removed.txt").exists()); + assert!(!rootfs.join("dir/old.txt").exists()); + assert_eq!( + fs::read_to_string(rootfs.join("dir/new.txt")).unwrap(), + "new" + ); + + let _ = fs::remove_dir_all(base); + } + + #[test] + fn apply_layer_dir_to_rootfs_preserves_lower_symlink_dirs() { + let base = unique_temp_dir(); + let rootfs = base.join("rootfs"); + let layer = base.join("layer"); + + fs::create_dir_all(rootfs.join("usr/bin")).unwrap(); + fs::write(rootfs.join("usr/bin/bash"), "bash").unwrap(); + std::os::unix::fs::symlink("usr/bin", rootfs.join("bin")).unwrap(); + + fs::create_dir_all(layer.join("bin")).unwrap(); + fs::write(layer.join("bin/foo"), "foo").unwrap(); + + apply_layer_dir_to_rootfs(&layer, &rootfs).unwrap(); + + assert!( + fs::symlink_metadata(rootfs.join("bin")) + .unwrap() + .file_type() + .is_symlink(), + "lower /bin symlink should be preserved" + ); + assert_eq!( + fs::read_to_string(rootfs.join("usr/bin/bash")).unwrap(), + "bash" + ); + assert_eq!( + fs::read_to_string(rootfs.join("usr/bin/foo")).unwrap(), + "foo" + ); + + let _ = fs::remove_dir_all(base); + } + + #[test] + fn layer_compression_from_media_type_supports_common_formats() { + assert_eq!( + layer_compression_from_media_type("application/vnd.oci.image.layer.v1.tar").unwrap(), + LayerCompression::None + ); + assert_eq!( + layer_compression_from_media_type("application/vnd.oci.image.layer.v1.tar+gzip") + .unwrap(), + LayerCompression::Gzip + ); + assert_eq!( + layer_compression_from_media_type("application/vnd.oci.image.layer.v1.tar+zstd") + .unwrap(), + LayerCompression::Zstd + ); + } + + #[test] + fn build_guest_environment_includes_tls_paths_for_https_endpoint() { + let config = VmDriverConfig { + openshell_endpoint: "https://127.0.0.1:8443".to_string(), + ssh_handshake_secret: "secret".to_string(), + guest_tls_ca: Some(PathBuf::from("/host/ca.crt")), + guest_tls_cert: Some(PathBuf::from("/host/tls.crt")), + guest_tls_key: Some(PathBuf::from("/host/tls.key")), + ..Default::default() + }; + let sandbox = Sandbox { + id: "sandbox-123".to_string(), + name: "sandbox-123".to_string(), + spec: Some(SandboxSpec::default()), + ..Default::default() + }; + + let env = build_guest_environment(&sandbox, &config, None); + assert!(env.contains(&format!("OPENSHELL_TLS_CA={GUEST_TLS_CA_PATH}"))); + assert!(env.contains(&format!("OPENSHELL_TLS_CERT={GUEST_TLS_CERT_PATH}"))); + assert!(env.contains(&format!("OPENSHELL_TLS_KEY={GUEST_TLS_KEY_PATH}"))); + } + + #[test] + fn vm_driver_config_requires_tls_materials_for_https_endpoint() { + let config = VmDriverConfig { + openshell_endpoint: "https://127.0.0.1:8443".to_string(), + ..Default::default() + }; + let err = config + .tls_paths() + .expect_err("https endpoint should require TLS materials"); + assert!(err.contains("OPENSHELL_VM_TLS_CA")); + } + + #[tokio::test] + async fn delete_sandbox_keeps_registry_entry_when_cleanup_fails() { + let (events, _) = broadcast::channel(WATCH_BUFFER); + let driver = VmDriver { + config: VmDriverConfig::default(), + launcher_bin: PathBuf::from("openshell-driver-vm"), + registry: Arc::new(Mutex::new(HashMap::new())), + image_cache_lock: Arc::new(Mutex::new(())), + events, + gpu_inventory: None, + subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( + Ipv4Addr::new(10, 0, 128, 0), + 17, + ))), + }; + + let base = unique_temp_dir(); + std::fs::create_dir_all(&base).unwrap(); + let state_file = base.join("state-file"); + std::fs::write(&state_file, "not a directory").unwrap(); + + insert_test_record( + &driver, + "sandbox-123", + state_file.clone(), + spawn_exited_child(), + ) + .await; + + let err = driver + .delete_sandbox("sandbox-123", "sandbox-123") + .await + .expect_err("state dir cleanup should fail for a file path"); + assert!(err.message().contains("failed to remove state dir")); + assert!(driver.registry.lock().await.contains_key("sandbox-123")); + + let retry_state_dir = base.join("state-dir"); + std::fs::create_dir_all(&retry_state_dir).unwrap(); + { + let mut registry = driver.registry.lock().await; + let record = registry.get_mut("sandbox-123").unwrap(); + record.state_dir = retry_state_dir; + record.process = Arc::new(Mutex::new(VmProcess { + child: spawn_exited_child(), + deleting: false, + })); + } + + let response = driver + .delete_sandbox("sandbox-123", "sandbox-123") + .await + .expect("delete retry should succeed once cleanup works"); + assert!(response.deleted); + assert!(!driver.registry.lock().await.contains_key("sandbox-123")); + + let _ = std::fs::remove_dir_all(base); + } + + #[test] + fn validate_openshell_endpoint_accepts_loopback_hosts() { + validate_openshell_endpoint("http://127.0.0.1:8080") + .expect("ipv4 loopback should be allowed for TSI"); + validate_openshell_endpoint("http://localhost:8080") + .expect("localhost should be allowed for TSI"); + validate_openshell_endpoint("http://[::1]:8080") + .expect("ipv6 loopback should be allowed for TSI"); + } + + #[test] + fn validate_openshell_endpoint_rejects_unspecified_hosts() { + let err = validate_openshell_endpoint("http://0.0.0.0:8080") + .expect_err("unspecified endpoint should fail"); + assert!(err.contains("not reachable from sandbox VMs")); + } + + #[test] + fn validate_openshell_endpoint_accepts_host_gateway() { + validate_openshell_endpoint("http://host.containers.internal:8080") + .expect("guest-reachable host alias should be accepted"); + validate_openshell_endpoint("http://192.168.127.1:8080") + .expect("gateway IP should be accepted"); + validate_openshell_endpoint(&format!("http://{OPENSHELL_HOST_GATEWAY_ALIAS}:8080")) + .expect("openshell host alias should be accepted"); + validate_openshell_endpoint("https://gateway.internal:8443") + .expect("dns endpoint should be accepted"); + } + + #[test] + fn prepare_exported_rootfs_archive_rewrites_docker_exported_rootfs() { + let base = unique_temp_dir(); + let source_rootfs = base.join("source-rootfs"); + let exported_rootfs = base.join("exported-rootfs.tar"); + let prepared_rootfs = base.join("prepared-rootfs"); + let prepared_archive = base.join("prepared-rootfs.tar"); + let extracted = base.join("extracted"); + + for path in [ + "bin/bash", + "bin/mount", + "bin/sed", + "sbin/ip", + "opt/openshell/bin/openshell-sandbox", + "usr/local/bin/k3s", + ] { + let path = source_rootfs.join(path); + fs::create_dir_all(path.parent().unwrap()).unwrap(); + fs::write(path, "").unwrap(); + } + fs::create_dir_all(source_rootfs.join("opt/openshell/manifests")).unwrap(); + fs::write(source_rootfs.join("opt/openshell/manifests/old.yaml"), "").unwrap(); + + create_rootfs_archive_from_dir(&source_rootfs, &exported_rootfs).unwrap(); + prepare_exported_rootfs_archive( + "openshell/sandbox-from:123", + "sha256:local-image", + &exported_rootfs, + &prepared_rootfs, + &prepared_archive, + ) + .unwrap(); + extract_rootfs_archive_to(&prepared_archive, &extracted).unwrap(); + + assert!(extracted.join("srv/openshell-vm-sandbox-init.sh").is_file()); + assert!( + extracted + .join("opt/openshell/bin/openshell-sandbox") + .is_file() + ); + assert!(!extracted.join("usr/local/bin/k3s").exists()); + assert!(!extracted.join("opt/openshell/manifests").exists()); + assert_eq!( + fs::read_to_string(extracted.join("opt/openshell/.rootfs-type")).unwrap(), + "sandbox\n" + ); + assert!( + fs::read_to_string(extracted.join(".openshell-rootfs-variant")) + .unwrap() + .contains("sha256:local-image") + ); + + let _ = fs::remove_dir_all(base); + } + + #[test] + fn sanitize_image_identity_rewrites_path_separators() { + assert_eq!( + sanitize_image_identity("sha256:abc/def@ghi"), + "sha256-abc-def-ghi" + ); + } + + #[tokio::test] + async fn prepare_guest_tls_materials_copies_bundle_into_rootfs() { + let base = unique_temp_dir(); + let source_dir = base.join("source"); + let rootfs = base.join("rootfs"); + std::fs::create_dir_all(&source_dir).unwrap(); + std::fs::create_dir_all(&rootfs).unwrap(); + + let ca = source_dir.join("ca.crt"); + let cert = source_dir.join("tls.crt"); + let key = source_dir.join("tls.key"); + std::fs::write(&ca, "ca").unwrap(); + std::fs::write(&cert, "cert").unwrap(); + std::fs::write(&key, "key").unwrap(); + + prepare_guest_tls_materials( + &rootfs, + &VmDriverTlsPaths { + ca: ca.clone(), + cert: cert.clone(), + key: key.clone(), + }, + ) + .await + .unwrap(); + + let guest_dir = rootfs.join(GUEST_TLS_DIR.trim_start_matches('/')); + assert_eq!( + std::fs::read_to_string(guest_dir.join("ca.crt")).unwrap(), + "ca" + ); + assert_eq!( + std::fs::read_to_string(guest_dir.join("tls.crt")).unwrap(), + "cert" + ); + assert_eq!( + std::fs::read_to_string(guest_dir.join("tls.key")).unwrap(), + "key" + ); + let key_mode = std::fs::metadata(guest_dir.join("tls.key")) + .unwrap() + .permissions() + .mode() + & 0o777; + assert_eq!(key_mode, 0o600); + + let _ = std::fs::remove_dir_all(base); + } + + #[test] + fn subnet_allocator_assigns_and_releases() { + let mut alloc = SubnetAllocator::new(Ipv4Addr::new(10, 0, 128, 0), 17); + let s1 = alloc.allocate("sandbox-1").unwrap(); + assert_eq!(s1.host_ip, Ipv4Addr::new(10, 0, 128, 1)); + assert_eq!(s1.guest_ip, Ipv4Addr::new(10, 0, 128, 2)); + assert_eq!(s1.prefix_len, 30); + + let s2 = alloc.allocate("sandbox-2").unwrap(); + assert_ne!(s1.host_ip, s2.host_ip); + + alloc.release("sandbox-1"); + let s3 = alloc.allocate("sandbox-3").unwrap(); + assert!(s3.host_ip != s2.host_ip); + } + + #[test] + fn tap_device_name_fits_ifnamsiz() { + let name = tap_device_name("sandbox-abc-def-ghi"); + assert!(name.len() <= 15); + assert!(name.starts_with("vmtap-")); + } + + #[test] + fn mac_address_is_locally_administered() { + let mac = mac_from_sandbox_id("test-sandbox"); + assert_eq!(mac[0] & 0x02, 0x02); + assert_eq!(mac[0] & 0x01, 0x00); + } + + #[test] + fn vsock_cid_monotonically_increases() { + let cid1 = allocate_vsock_cid(); + let cid2 = allocate_vsock_cid(); + assert!(cid2 > cid1); + } + + fn unique_temp_dir() -> PathBuf { + static COUNTER: AtomicU64 = AtomicU64::new(0); + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos(); + let suffix = COUNTER.fetch_add(1, Ordering::Relaxed); + std::env::temp_dir().join(format!( + "openshell-vm-driver-test-{}-{nanos}-{suffix}", + std::process::id() + )) + } + + fn spawn_exited_child() -> Child { + Command::new("sh") + .arg("-c") + .arg("exit 0") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .unwrap() + } + + async fn insert_test_record( + driver: &VmDriver, + sandbox_id: &str, + state_dir: PathBuf, + child: Child, + ) { + let sandbox = Sandbox { + id: sandbox_id.to_string(), + name: sandbox_id.to_string(), + ..Default::default() + }; + let process = Arc::new(Mutex::new(VmProcess { + child, + deleting: false, + })); + + let mut registry = driver.registry.lock().await; + registry.insert( + sandbox_id.to_string(), + SandboxRecord { + snapshot: sandbox, + state_dir, + process, + gpu_bdf: None, + }, + ); + } +} diff --git a/crates/openshell-driver-vm/src/embedded_runtime.rs b/crates/openshell-driver-vm/src/embedded_runtime.rs new file mode 100644 index 0000000000..70626edd98 --- /dev/null +++ b/crates/openshell-driver-vm/src/embedded_runtime.rs @@ -0,0 +1,182 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Embedded libkrun runtime resources for the VM driver. + +use std::fs; +use std::path::{Path, PathBuf}; + +#[cfg(all(target_os = "macos", target_arch = "aarch64"))] +mod resources { + pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.dylib.zst")); + pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.5.dylib.zst")); + pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst")); + pub const LIBKRUN_NAME: &str = "libkrun.dylib"; + pub const LIBKRUNFW_NAME: &str = "libkrunfw.5.dylib"; +} + +#[cfg(all(target_os = "linux", target_arch = "aarch64"))] +mod resources { + pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.so.zst")); + pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.so.5.zst")); + pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst")); + pub const LIBKRUN_NAME: &str = "libkrun.so"; + pub const LIBKRUNFW_NAME: &str = "libkrunfw.so.5"; +} + +#[cfg(all(target_os = "linux", target_arch = "x86_64"))] +mod resources { + pub const LIBKRUN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrun.so.zst")); + pub const LIBKRUNFW: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/libkrunfw.so.5.zst")); + pub const GVPROXY: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/gvproxy.zst")); + pub const LIBKRUN_NAME: &str = "libkrun.so"; + pub const LIBKRUNFW_NAME: &str = "libkrunfw.so.5"; +} + +#[cfg(not(any( + all(target_os = "macos", target_arch = "aarch64"), + all(target_os = "linux", target_arch = "aarch64"), + all(target_os = "linux", target_arch = "x86_64"), +)))] +mod resources { + pub const LIBKRUN: &[u8] = &[]; + pub const LIBKRUNFW: &[u8] = &[]; + pub const GVPROXY: &[u8] = &[]; + pub const LIBKRUN_NAME: &str = "libkrun"; + pub const LIBKRUNFW_NAME: &str = "libkrunfw"; +} + +const VERSION: &str = env!("CARGO_PKG_VERSION"); + +pub fn ensure_runtime_extracted() -> Result { + if resources::LIBKRUN.is_empty() { + return Err( + "VM runtime not embedded for this platform. Supported: macOS ARM64, Linux ARM64, Linux x86_64" + .to_string(), + ); + } + + let cache_dir = runtime_cache_dir()?; + let version_marker = cache_dir.join(".version"); + let cache_key = runtime_cache_key(); + + if version_marker.exists() + && let Ok(cached_key) = fs::read_to_string(&version_marker) + && cached_key.trim() == cache_key + && validate_runtime_dir(&cache_dir).is_ok() + { + return Ok(cache_dir); + } + + cleanup_old_versions(&cache_dir)?; + + if cache_dir.exists() { + fs::remove_dir_all(&cache_dir) + .map_err(|e| format!("remove old runtime cache {}: {e}", cache_dir.display()))?; + } + fs::create_dir_all(&cache_dir) + .map_err(|e| format!("create runtime cache {}: {e}", cache_dir.display()))?; + + extract_resource(resources::LIBKRUN, &cache_dir.join(resources::LIBKRUN_NAME))?; + extract_resource( + resources::LIBKRUNFW, + &cache_dir.join(resources::LIBKRUNFW_NAME), + )?; + extract_resource(resources::GVPROXY, &cache_dir.join("gvproxy"))?; + + #[cfg(target_os = "macos")] + { + let unversioned = cache_dir.join("libkrunfw.dylib"); + if !unversioned.exists() { + std::os::unix::fs::symlink(resources::LIBKRUNFW_NAME, &unversioned) + .map_err(|e| format!("symlink {}: {e}", unversioned.display()))?; + } + } + + fs::write(&version_marker, cache_key) + .map_err(|e| format!("write runtime marker {}: {e}", version_marker.display()))?; + + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + fs::set_permissions(cache_dir.join("gvproxy"), fs::Permissions::from_mode(0o755)) + .map_err(|e| format!("chmod gvproxy: {e}"))?; + } + + Ok(cache_dir) +} + +pub fn validate_runtime_dir(dir: &Path) -> Result<(), String> { + let libkrun = dir.join(resources::LIBKRUN_NAME); + let libkrunfw = dir.join(resources::LIBKRUNFW_NAME); + let gvproxy = dir.join("gvproxy"); + + for path in [&libkrun, &libkrunfw, &gvproxy] { + if !path.is_file() { + return Err(format!("missing runtime file: {}", path.display())); + } + let size = fs::metadata(path).map_or(0, |m| m.len()); + if size == 0 { + return Err(format!("runtime file is empty (stub): {}", path.display())); + } + } + + Ok(()) +} + +fn runtime_cache_key() -> String { + let mut fp: u64 = 0; + for (index, chunk) in [resources::LIBKRUN, resources::LIBKRUNFW] + .into_iter() + .chain(std::iter::once(resources::GVPROXY)) + .enumerate() + { + let sample = &chunk[..chunk.len().min(64)]; + let mut word: u64 = 0; + for (offset, byte) in sample.iter().enumerate() { + word ^= u64::from(*byte) << ((offset % 8) * 8); + } + let index_u32 = u32::try_from(index).unwrap_or(u32::MAX); + fp ^= word.rotate_left(index_u32 * 13 + 7); + fp ^= (chunk.len() as u64).rotate_left(index_u32 * 17 + 3); + } + format!("{VERSION}-{fp:016x}") +} + +fn runtime_cache_dir() -> Result { + let base = + openshell_core::paths::xdg_data_dir().map_err(|e| format!("resolve XDG data dir: {e}"))?; + Ok(base.join("openshell").join("vm-runtime").join(VERSION)) +} + +fn runtime_cache_base() -> Result { + let base = + openshell_core::paths::xdg_data_dir().map_err(|e| format!("resolve XDG data dir: {e}"))?; + Ok(base.join("openshell").join("vm-runtime")) +} + +fn cleanup_old_versions(current_dir: &Path) -> Result<(), String> { + let base = runtime_cache_base()?; + if !base.exists() { + return Ok(()); + } + + let entries = fs::read_dir(&base).map_err(|e| format!("read {}: {e}", base.display()))?; + for entry in entries.filter_map(Result::ok) { + let path = entry.path(); + if path.is_dir() && !current_dir.starts_with(&path) && path != current_dir { + let _ = fs::remove_dir_all(&path); + } + } + Ok(()) +} + +fn extract_resource(compressed: &[u8], dest: &Path) -> Result<(), String> { + if compressed.is_empty() { + return Err(format!("embedded resource is empty: {}", dest.display())); + } + + let decompressed = + zstd::decode_all(compressed).map_err(|e| format!("decompress {}: {e}", dest.display()))?; + fs::write(dest, decompressed).map_err(|e| format!("write {}: {e}", dest.display())) +} diff --git a/crates/openshell-driver-vm/src/ffi.rs b/crates/openshell-driver-vm/src/ffi.rs new file mode 100644 index 0000000000..db5d3ec108 --- /dev/null +++ b/crates/openshell-driver-vm/src/ffi.rs @@ -0,0 +1,209 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Minimal runtime-loaded bindings for the libkrun C API used by the VM driver. + +#![allow(unsafe_code)] + +use std::ffi::{CStr, CString}; +use std::path::{Path, PathBuf}; +use std::sync::OnceLock; + +use libc::c_char; +use libloading::Library; + +use crate::runtime::validate_runtime_dir; + +pub const KRUN_LOG_TARGET_DEFAULT: i32 = -1; +pub const KRUN_LOG_LEVEL_OFF: u32 = 0; +pub const KRUN_LOG_LEVEL_ERROR: u32 = 1; +pub const KRUN_LOG_LEVEL_WARN: u32 = 2; +pub const KRUN_LOG_LEVEL_INFO: u32 = 3; +pub const KRUN_LOG_LEVEL_DEBUG: u32 = 4; +pub const KRUN_LOG_LEVEL_TRACE: u32 = 5; +pub const KRUN_LOG_STYLE_AUTO: u32 = 0; +pub const KRUN_LOG_OPTION_NO_ENV: u32 = 1; + +type KrunInitLog = + unsafe extern "C" fn(target_fd: i32, level: u32, style: u32, options: u32) -> i32; +type KrunCreateCtx = unsafe extern "C" fn() -> i32; +type KrunFreeCtx = unsafe extern "C" fn(ctx_id: u32) -> i32; +type KrunSetVmConfig = unsafe extern "C" fn(ctx_id: u32, num_vcpus: u8, ram_mib: u32) -> i32; +type KrunSetRoot = unsafe extern "C" fn(ctx_id: u32, root_path: *const c_char) -> i32; +type KrunSetWorkdir = unsafe extern "C" fn(ctx_id: u32, workdir_path: *const c_char) -> i32; +type KrunSetExec = unsafe extern "C" fn( + ctx_id: u32, + exec_path: *const c_char, + argv: *const *const c_char, + envp: *const *const c_char, +) -> i32; +type KrunSetConsoleOutput = unsafe extern "C" fn(ctx_id: u32, filepath: *const c_char) -> i32; +type KrunStartEnter = unsafe extern "C" fn(ctx_id: u32) -> i32; +type KrunDisableImplicitVsock = unsafe extern "C" fn(ctx_id: u32) -> i32; +type KrunAddVsock = unsafe extern "C" fn(ctx_id: u32, tsi_features: u32) -> i32; +#[cfg(target_os = "macos")] +type KrunAddNetUnixgram = unsafe extern "C" fn( + ctx_id: u32, + c_path: *const c_char, + fd: i32, + c_mac: *const u8, + features: u32, + flags: u32, +) -> i32; +type KrunAddNetUnixstream = unsafe extern "C" fn( + ctx_id: u32, + c_path: *const c_char, + fd: i32, + c_mac: *const u8, + features: u32, + flags: u32, +) -> i32; + +// Field names mirror the libkrun C API symbol names (`krun_*`); preserving +// the prefix keeps the FFI binding 1:1 with the upstream library. +#[allow(clippy::struct_field_names)] +pub struct LibKrun { + pub krun_init_log: KrunInitLog, + pub krun_create_ctx: KrunCreateCtx, + pub krun_free_ctx: KrunFreeCtx, + pub krun_set_vm_config: KrunSetVmConfig, + pub krun_set_root: KrunSetRoot, + pub krun_set_workdir: KrunSetWorkdir, + pub krun_set_exec: KrunSetExec, + pub krun_set_console_output: KrunSetConsoleOutput, + pub krun_start_enter: KrunStartEnter, + pub krun_disable_implicit_vsock: KrunDisableImplicitVsock, + pub krun_add_vsock: KrunAddVsock, + #[cfg(target_os = "macos")] + pub krun_add_net_unixgram: KrunAddNetUnixgram, + #[allow(dead_code)] // Used on Linux when gvproxy runs in qemu/unixstream mode. + pub krun_add_net_unixstream: KrunAddNetUnixstream, +} + +static LIBKRUN: OnceLock = OnceLock::new(); + +pub fn libkrun(runtime_dir: &Path) -> Result<&'static LibKrun, String> { + if let Some(lib) = LIBKRUN.get() { + return Ok(lib); + } + + validate_runtime_dir(runtime_dir)?; + let loaded = LibKrun::load(runtime_dir)?; + let _ = LIBKRUN.set(loaded); + Ok(LIBKRUN.get().expect("libkrun should be initialized")) +} + +pub fn required_runtime_lib_name() -> &'static str { + #[cfg(target_os = "macos")] + { + "libkrun.dylib" + } + #[cfg(not(target_os = "macos"))] + { + "libkrun.so" + } +} + +impl LibKrun { + fn load(runtime_dir: &Path) -> Result { + let libkrun_path = runtime_dir.join(required_runtime_lib_name()); + preload_runtime_support_libraries(runtime_dir)?; + + let library = Box::leak(Box::new(unsafe { + Library::new(&libkrun_path) + .map_err(|e| format!("load libkrun from {}: {e}", libkrun_path.display()))? + })); + + Ok(Self { + krun_init_log: load_symbol(library, b"krun_init_log\0", &libkrun_path)?, + krun_create_ctx: load_symbol(library, b"krun_create_ctx\0", &libkrun_path)?, + krun_free_ctx: load_symbol(library, b"krun_free_ctx\0", &libkrun_path)?, + krun_set_vm_config: load_symbol(library, b"krun_set_vm_config\0", &libkrun_path)?, + krun_set_root: load_symbol(library, b"krun_set_root\0", &libkrun_path)?, + krun_set_workdir: load_symbol(library, b"krun_set_workdir\0", &libkrun_path)?, + krun_set_exec: load_symbol(library, b"krun_set_exec\0", &libkrun_path)?, + krun_set_console_output: load_symbol( + library, + b"krun_set_console_output\0", + &libkrun_path, + )?, + krun_start_enter: load_symbol(library, b"krun_start_enter\0", &libkrun_path)?, + krun_disable_implicit_vsock: load_symbol( + library, + b"krun_disable_implicit_vsock\0", + &libkrun_path, + )?, + krun_add_vsock: load_symbol(library, b"krun_add_vsock\0", &libkrun_path)?, + #[cfg(target_os = "macos")] + krun_add_net_unixgram: load_symbol(library, b"krun_add_net_unixgram\0", &libkrun_path)?, + krun_add_net_unixstream: load_symbol( + library, + b"krun_add_net_unixstream\0", + &libkrun_path, + )?, + }) + } +} + +fn preload_runtime_support_libraries(runtime_dir: &Path) -> Result, String> { + let entries = std::fs::read_dir(runtime_dir) + .map_err(|e| format!("read {}: {e}", runtime_dir.display()))?; + + let mut support_libs: Vec = entries + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| { + path.file_name() + .and_then(|name| name.to_str()) + .is_some_and(|name| { + #[cfg(target_os = "macos")] + { + name.starts_with("libkrunfw") + && Path::new(name) + .extension() + .is_some_and(|ext| ext.eq_ignore_ascii_case("dylib")) + } + #[cfg(not(target_os = "macos"))] + { + name.starts_with("libkrunfw") && name.contains(".so") + } + }) + }) + .collect(); + + support_libs.sort(); + for path in &support_libs { + let path_cstr = CString::new(path.to_string_lossy().as_bytes()) + .map_err(|e| format!("invalid support library path {}: {e}", path.display()))?; + let handle = + unsafe { libc::dlopen(path_cstr.as_ptr(), libc::RTLD_NOW | libc::RTLD_GLOBAL) }; + if handle.is_null() { + let error = unsafe { + let err = libc::dlerror(); + if err.is_null() { + "unknown dlopen error".to_string() + } else { + CStr::from_ptr(err).to_string_lossy().into_owned() + } + }; + return Err(format!( + "preload runtime support library {}: {error}", + path.display() + )); + } + } + + Ok(support_libs) +} + +fn load_symbol(library: &'static Library, name: &[u8], path: &Path) -> Result { + unsafe { + library.get::(name).map(|symbol| *symbol).map_err(|e| { + format!( + "load symbol {} from {}: {e}", + String::from_utf8_lossy(name).trim_end_matches('\0'), + path.display() + ) + }) + } +} diff --git a/crates/openshell-driver-vm/src/gpu.rs b/crates/openshell-driver-vm/src/gpu.rs new file mode 100644 index 0000000000..dc5883b5ba --- /dev/null +++ b/crates/openshell-driver-vm/src/gpu.rs @@ -0,0 +1,323 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use openshell_vfio::{ + GpuBindGuard, GpuBindState, GpuBinding, GpuInfo, SysfsRoot, prepare_gpu_for_passthrough, + probe_host_nvidia_vfio_readiness, reconcile_stale_bindings, validate_bdf, +}; +use std::collections::HashMap; +use std::net::Ipv4Addr; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU32, Ordering}; + +/// Tracks available GPUs and their assignment to sandboxes. +pub struct GpuInventory { + slots: Vec, + sysfs: SysfsRoot, + state_path: PathBuf, +} + +struct GpuSlot { + info: GpuInfo, + assigned_to: Option, + bind_guard: Option, +} + +impl GpuInventory { + pub fn new(sysfs: SysfsRoot, state_dir: &Path) -> Self { + let state_path = state_dir.join("gpu-bindings.json"); + + let restored = reconcile_stale_bindings(&sysfs, &state_path); + for bdf in &restored { + tracing::info!(bdf = %bdf, "restored stale GPU binding from previous crash"); + } + + let gpus = probe_host_nvidia_vfio_readiness(&sysfs); + let slots = gpus + .into_iter() + .map(|info| GpuSlot { + info, + assigned_to: None, + bind_guard: None, + }) + .collect(); + + Self { + slots, + sysfs, + state_path, + } + } + + pub fn gpu_count(&self) -> u32 { + u32::try_from(self.slots.len()).unwrap_or(u32::MAX) + } + + pub fn available_count(&self) -> u32 { + u32::try_from( + self.slots + .iter() + .filter(|s| s.assigned_to.is_none()) + .count(), + ) + .unwrap_or(u32::MAX) + } + + /// Assign a GPU to a sandbox. Returns the assignment details including BDF. + pub fn assign(&mut self, sandbox_id: &str, gpu_device: &str) -> Result { + let slot_idx = if gpu_device.is_empty() { + self.slots + .iter() + .position(|s| s.assigned_to.is_none()) + .ok_or_else(|| "all GPUs are currently assigned to other sandboxes".to_string())? + } else if let Ok(idx) = gpu_device.parse::() { + if idx >= self.slots.len() { + return Err(format!( + "GPU index {idx} out of range (have {} GPUs)", + self.slots.len() + )); + } + if self.slots[idx].assigned_to.is_some() { + return Err(format!( + "GPU at index {idx} ({}) is already assigned to another sandbox", + self.slots[idx].info.bdf + )); + } + idx + } else { + validate_bdf(gpu_device).map_err(|e| e.to_string())?; + let idx = self + .slots + .iter() + .position(|s| s.info.bdf == gpu_device) + .ok_or_else(|| format!("GPU {gpu_device} not found in inventory"))?; + if self.slots[idx].assigned_to.is_some() { + return Err(format!( + "GPU {gpu_device} is already assigned to another sandbox" + )); + } + idx + }; + + let bdf = self.slots[slot_idx].info.bdf.clone(); + let guard = prepare_gpu_for_passthrough(&self.sysfs, &bdf) + .map_err(|e| format!("failed to prepare GPU {bdf} for passthrough: {e}"))?; + + self.slots[slot_idx].assigned_to = Some(sandbox_id.to_string()); + self.slots[slot_idx].bind_guard = Some(guard); + self.persist_state(); + + Ok(GpuAssignment { + bdf, + name: self.slots[slot_idx].info.name.clone(), + iommu_group: self.slots[slot_idx].info.iommu_group, + }) + } + + /// Release a GPU assignment. The `GpuBindGuard` is dropped, restoring the GPU. + pub fn release(&mut self, sandbox_id: &str) { + if let Some(slot) = self + .slots + .iter_mut() + .find(|s| s.assigned_to.as_deref() == Some(sandbox_id)) + { + let bdf = slot.info.bdf.clone(); + slot.assigned_to = None; + slot.bind_guard.take(); + self.persist_state(); + tracing::info!(bdf = %bdf, sandbox_id = %sandbox_id, "released GPU assignment"); + } + } + + fn persist_state(&self) { + let bindings: Vec = self + .slots + .iter() + .filter_map(|s| { + s.assigned_to.as_ref().map(|id| GpuBinding { + bdf: s.info.bdf.clone(), + sandbox_id: id.clone(), + bound_at_ms: std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |d| i64::try_from(d.as_millis()).unwrap_or(i64::MAX)), + }) + }) + .collect(); + let state = GpuBindState { bindings }; + if let Err(err) = state.save(&self.state_path) { + tracing::warn!(error = %err, "failed to persist GPU bind state"); + } + } +} + +pub struct GpuAssignment { + pub bdf: String, + pub name: String, + pub iommu_group: u32, +} + +// --------------------------------------------------------------------------- +// Subnet allocation for per-sandbox TAP networking +// --------------------------------------------------------------------------- + +/// Allocates /30 subnets from a pool for per-sandbox TAP networking. +pub struct SubnetAllocator { + base: Ipv4Addr, + prefix_len: u8, + next_offset: u32, + allocated: HashMap, +} + +pub struct SubnetAllocation { + pub host_ip: Ipv4Addr, + pub guest_ip: Ipv4Addr, + pub prefix_len: u8, + pub offset: u32, +} + +static NEXT_VSOCK_CID: AtomicU32 = AtomicU32::new(3); + +impl SubnetAllocator { + pub fn new(base: Ipv4Addr, prefix_len: u8) -> Self { + Self { + base, + prefix_len, + next_offset: 0, + allocated: HashMap::new(), + } + } + + pub fn allocate(&mut self, sandbox_id: &str) -> Result { + let pool_size = 1u32 << (32 - self.prefix_len); + let max_subnets = pool_size / 4; + + if u32::try_from(self.allocated.len()).unwrap_or(u32::MAX) >= max_subnets { + return Err("subnet pool exhausted".to_string()); + } + + while self + .allocated + .values() + .any(|a| a.offset == self.next_offset) + { + self.next_offset = (self.next_offset + 1) % max_subnets; + } + + let base_u32 = u32::from(self.base); + let subnet_base = base_u32 + (self.next_offset * 4); + let host_ip = Ipv4Addr::from(subnet_base + 1); + let guest_ip = Ipv4Addr::from(subnet_base + 2); + + let allocation = SubnetAllocation { + host_ip, + guest_ip, + prefix_len: 30, + offset: self.next_offset, + }; + + self.allocated.insert(sandbox_id.to_string(), allocation); + self.next_offset = (self.next_offset + 1) % max_subnets; + + let alloc = &self.allocated[sandbox_id]; + Ok(SubnetAllocation { + host_ip: alloc.host_ip, + guest_ip: alloc.guest_ip, + prefix_len: alloc.prefix_len, + offset: alloc.offset, + }) + } + + pub fn release(&mut self, sandbox_id: &str) { + self.allocated.remove(sandbox_id); + } +} + +pub fn allocate_vsock_cid() -> u32 { + NEXT_VSOCK_CID.fetch_add(1, Ordering::Relaxed) +} + +/// Generate a locally-administered MAC from sandbox ID using FNV-1a. +pub fn mac_from_sandbox_id(sandbox_id: &str) -> [u8; 6] { + let mut hash: u64 = 0xcbf2_9ce4_8422_2325; + for byte in sandbox_id.as_bytes() { + hash ^= u64::from(*byte); + hash = hash.wrapping_mul(0x0000_0100_0000_01b3); + } + let bytes = hash.to_le_bytes(); + let mut mac = [bytes[0], bytes[1], bytes[2], bytes[3], bytes[4], bytes[5]]; + mac[0] = (mac[0] & 0xFE) | 0x02; + mac +} + +/// TAP device name from sandbox ID (fits `IFNAMSIZ=16`). +pub fn tap_device_name(sandbox_id: &str) -> String { + let mut end = sandbox_id.len().min(8); + // Walk back to a UTF-8 char boundary (str::floor_char_boundary requires + // Rust 1.91 — we still build on older toolchains). + while end > 0 && !sandbox_id.is_char_boundary(end) { + end -= 1; + } + let prefix = &sandbox_id[..end]; + format!("vmtap-{prefix}") +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn subnet_allocator_assigns_sequential_blocks() { + let mut alloc = SubnetAllocator::new(Ipv4Addr::new(10, 0, 128, 0), 17); + + let s1 = alloc.allocate("sandbox-1").unwrap(); + assert_eq!(s1.host_ip, Ipv4Addr::new(10, 0, 128, 1)); + assert_eq!(s1.guest_ip, Ipv4Addr::new(10, 0, 128, 2)); + assert_eq!(s1.prefix_len, 30); + + let s2 = alloc.allocate("sandbox-2").unwrap(); + assert_eq!(s2.host_ip, Ipv4Addr::new(10, 0, 128, 5)); + assert_eq!(s2.guest_ip, Ipv4Addr::new(10, 0, 128, 6)); + } + + #[test] + fn subnet_allocator_recycles_after_release() { + let mut alloc = SubnetAllocator::new(Ipv4Addr::new(10, 0, 128, 0), 17); + + let _s1 = alloc.allocate("sandbox-1").unwrap(); + let _s2 = alloc.allocate("sandbox-2").unwrap(); + alloc.release("sandbox-1"); + + let s3 = alloc.allocate("sandbox-3").unwrap(); + assert_eq!(s3.host_ip, Ipv4Addr::new(10, 0, 128, 9)); + } + + #[test] + fn tap_device_name_truncates_long_ids() { + assert_eq!(tap_device_name("abc"), "vmtap-abc"); + assert_eq!(tap_device_name("abcdefghijklmnop"), "vmtap-abcdefgh"); + } + + #[test] + fn mac_from_sandbox_id_sets_locally_administered_bit() { + let mac = mac_from_sandbox_id("sandbox-123"); + assert_eq!(mac[0] & 0x02, 0x02, "locally-administered bit must be set"); + assert_eq!(mac[0] & 0x01, 0x00, "multicast bit must be clear"); + } + + #[test] + fn mac_from_sandbox_id_deterministic() { + let mac1 = mac_from_sandbox_id("sandbox-x"); + let mac2 = mac_from_sandbox_id("sandbox-x"); + assert_eq!(mac1, mac2); + + let mac3 = mac_from_sandbox_id("sandbox-y"); + assert_ne!(mac1, mac3); + } + + #[test] + fn vsock_cid_increments() { + let cid1 = allocate_vsock_cid(); + let cid2 = allocate_vsock_cid(); + assert_eq!(cid2, cid1 + 1); + } +} diff --git a/crates/openshell-driver-vm/src/lib.rs b/crates/openshell-driver-vm/src/lib.rs new file mode 100644 index 0000000000..194dde43c2 --- /dev/null +++ b/crates/openshell-driver-vm/src/lib.rs @@ -0,0 +1,16 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +pub mod driver; +mod embedded_runtime; +mod ffi; +pub mod gpu; +pub mod procguard; +mod rootfs; +mod runtime; + +pub use driver::{VmDriver, VmDriverConfig}; +pub use runtime::{ + VM_RUNTIME_DIR_ENV, VmBackend, VmLaunchConfig, cleanup_stale_tap_interfaces, + configured_runtime_dir, run_vm, +}; diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs new file mode 100644 index 0000000000..596e6c88da --- /dev/null +++ b/crates/openshell-driver-vm/src/main.rs @@ -0,0 +1,312 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use clap::Parser; +use miette::{IntoDiagnostic, Result}; +use openshell_core::VERSION; +use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; +#[cfg(target_os = "macos")] +use openshell_driver_vm::{VM_RUNTIME_DIR_ENV, configured_runtime_dir}; +use openshell_driver_vm::{VmBackend, VmDriver, VmDriverConfig, VmLaunchConfig, procguard, run_vm}; +use std::net::SocketAddr; +use std::path::PathBuf; +use tokio::net::UnixListener; +use tokio_stream::wrappers::UnixListenerStream; +use tracing::info; +use tracing_subscriber::EnvFilter; + +#[derive(Parser, Debug)] +#[command(name = "openshell-driver-vm")] +#[command(version = VERSION)] +struct Args { + #[arg(long, hide = true, default_value_t = false)] + internal_run_vm: bool, + + #[arg(long, hide = true)] + vm_rootfs: Option, + + #[arg(long, hide = true)] + vm_exec: Option, + + #[arg(long, hide = true, default_value = "/")] + vm_workdir: String, + + #[arg(long, hide = true)] + vm_env: Vec, + + #[arg(long, hide = true)] + vm_console_output: Option, + + #[arg(long, hide = true, default_value_t = 2)] + vm_vcpus: u8, + + #[arg(long, hide = true, default_value_t = 2048)] + vm_mem_mib: u32, + + #[arg(long, hide = true, default_value_t = 1)] + vm_krun_log_level: u32, + + #[arg( + long, + env = "OPENSHELL_COMPUTE_DRIVER_BIND", + default_value = "127.0.0.1:50061" + )] + bind_address: SocketAddr, + + #[arg(long, env = "OPENSHELL_COMPUTE_DRIVER_SOCKET")] + bind_socket: Option, + + #[arg(long, env = "OPENSHELL_LOG_LEVEL", default_value = "info")] + log_level: String, + + #[arg(long, env = "OPENSHELL_GRPC_ENDPOINT")] + openshell_endpoint: Option, + + #[arg(long, env = "OPENSHELL_SANDBOX_IMAGE", default_value = "")] + default_image: String, + + #[arg( + long, + env = "OPENSHELL_VM_DRIVER_STATE_DIR", + default_value = "target/openshell-vm-driver" + )] + state_dir: PathBuf, + + #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SECRET")] + ssh_handshake_secret: Option, + + #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SKEW_SECS", default_value_t = 300)] + ssh_handshake_skew_secs: u64, + + #[arg(long = "guest-tls-ca", env = "OPENSHELL_VM_TLS_CA")] + guest_tls_ca: Option, + + #[arg(long = "guest-tls-cert", env = "OPENSHELL_VM_TLS_CERT")] + guest_tls_cert: Option, + + #[arg(long = "guest-tls-key", env = "OPENSHELL_VM_TLS_KEY")] + guest_tls_key: Option, + + #[arg(long, env = "OPENSHELL_VM_KRUN_LOG_LEVEL", default_value_t = 1)] + krun_log_level: u32, + + #[arg(long, env = "OPENSHELL_VM_DRIVER_VCPUS", default_value_t = 2)] + vcpus: u8, + + #[arg(long, env = "OPENSHELL_VM_DRIVER_MEM_MIB", default_value_t = 2048)] + mem_mib: u32, + + #[arg(long, env = "OPENSHELL_VM_GPU")] + gpu: bool, + + #[arg(long, env = "OPENSHELL_VM_GPU_MEM_MIB", default_value_t = 8192)] + gpu_mem_mib: u32, + + #[arg(long, env = "OPENSHELL_VM_GPU_VCPUS", default_value_t = 4)] + gpu_vcpus: u8, + + #[arg(long, hide = true)] + vm_backend: Option, + + #[arg(long, hide = true)] + vm_gpu_bdf: Option, + + #[arg(long, hide = true)] + vm_tap_device: Option, + + #[arg(long, hide = true)] + vm_guest_ip: Option, + + #[arg(long, hide = true)] + vm_host_ip: Option, + + #[arg(long, hide = true)] + vm_vsock_cid: Option, + + #[arg(long, hide = true)] + vm_guest_mac: Option, + + #[arg(long, hide = true)] + vm_gateway_port: Option, +} + +#[tokio::main] +async fn main() -> Result<()> { + let args = Args::parse(); + if args.internal_run_vm { + // We intentionally defer procguard arming until `run_vm()` so + // that the only arm is the one that knows how to clean up + // gvproxy. Racing two watchers against the same parent-death + // event causes the bare arm's `exit(1)` to win, skipping the + // gvproxy cleanup and leaking the helper. The risk window + // before `run_vm` arms procguard is ~a few syscalls long + // (`build_vm_launch_config`, `configured_runtime_dir`), which + // is negligible next to the parent gRPC server's uptime. + maybe_reexec_internal_vm_with_runtime_env()?; + let config = build_vm_launch_config(&args).map_err(|err| miette::miette!("{err}"))?; + run_vm(&config).map_err(|err| miette::miette!("{err}"))?; + return Ok(()); + } + + tracing_subscriber::fmt() + .with_env_filter( + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)), + ) + .init(); + + // Arm procguard so that if the gateway is killed (SIGKILL or crash) + // we also die. Without this the driver is reparented to init and + // keeps its per-sandbox VM launchers alive forever. Launchers have + // their own procguards (armed in `run_vm`) which cascade cleanup of + // gvproxy and the libkrun worker the moment this driver exits. + if let Err(err) = procguard::die_with_parent() { + tracing::warn!( + error = %err, + "procguard arm failed; gateway crashes may orphan this driver" + ); + } + + let driver = VmDriver::new(VmDriverConfig { + openshell_endpoint: args + .openshell_endpoint + .ok_or_else(|| miette::miette!("OPENSHELL_GRPC_ENDPOINT is required"))?, + state_dir: args.state_dir, + launcher_bin: None, + default_image: args.default_image, + ssh_handshake_secret: args.ssh_handshake_secret.unwrap_or_default(), + ssh_handshake_skew_secs: args.ssh_handshake_skew_secs, + log_level: args.log_level, + krun_log_level: args.krun_log_level, + vcpus: args.vcpus, + mem_mib: args.mem_mib, + guest_tls_ca: args.guest_tls_ca, + guest_tls_cert: args.guest_tls_cert, + guest_tls_key: args.guest_tls_key, + gpu_enabled: args.gpu, + gpu_mem_mib: args.gpu_mem_mib, + gpu_vcpus: args.gpu_vcpus, + }) + .await + .map_err(|err| miette::miette!("{err}"))?; + + if let Some(socket_path) = args.bind_socket { + if let Some(parent) = socket_path.parent() { + std::fs::create_dir_all(parent).into_diagnostic()?; + } + match std::fs::remove_file(&socket_path) { + Ok(()) => {} + Err(err) if err.kind() == std::io::ErrorKind::NotFound => {} + Err(err) => return Err(err).into_diagnostic(), + } + + info!(socket = %socket_path.display(), "Starting vm compute driver"); + let listener = UnixListener::bind(&socket_path).into_diagnostic()?; + let result = tonic::transport::Server::builder() + .add_service(ComputeDriverServer::new(driver)) + .serve_with_incoming(UnixListenerStream::new(listener)) + .await + .into_diagnostic(); + let _ = std::fs::remove_file(&socket_path); + result + } else { + info!(address = %args.bind_address, "Starting vm compute driver"); + tonic::transport::Server::builder() + .add_service(ComputeDriverServer::new(driver)) + .serve(args.bind_address) + .await + .into_diagnostic() + } +} + +fn build_vm_launch_config(args: &Args) -> std::result::Result { + let rootfs = args + .vm_rootfs + .clone() + .ok_or_else(|| "--vm-rootfs is required in internal VM mode".to_string())?; + let exec_path = args + .vm_exec + .clone() + .ok_or_else(|| "--vm-exec is required in internal VM mode".to_string())?; + let console_output = args + .vm_console_output + .clone() + .ok_or_else(|| "--vm-console-output is required in internal VM mode".to_string())?; + + let backend = match args.vm_backend.as_deref() { + Some("qemu") => VmBackend::Qemu, + Some("libkrun") | None => VmBackend::Libkrun, + Some(other) => return Err(format!("unknown VM backend: {other}")), + }; + + Ok(VmLaunchConfig { + rootfs, + vcpus: args.vm_vcpus, + mem_mib: args.vm_mem_mib, + exec_path, + args: Vec::new(), + env: args.vm_env.clone(), + workdir: args.vm_workdir.clone(), + log_level: args.vm_krun_log_level, + console_output, + backend, + gpu_bdf: args.vm_gpu_bdf.clone(), + tap_device: args.vm_tap_device.clone(), + guest_ip: args.vm_guest_ip.clone(), + host_ip: args.vm_host_ip.clone(), + vsock_cid: args.vm_vsock_cid, + guest_mac: args.vm_guest_mac.clone(), + gateway_port: args.vm_gateway_port, + }) +} + +#[cfg(target_os = "macos")] +fn maybe_reexec_internal_vm_with_runtime_env() -> Result<()> { + use std::os::unix::process::CommandExt as _; + + const REEXEC_ENV: &str = "__OPENSHELL_DRIVER_VM_REEXEC"; + + if std::env::var_os(REEXEC_ENV).is_some() { + return Ok(()); + } + + let runtime_dir = configured_runtime_dir().map_err(|err| miette::miette!("{err}"))?; + let runtime_str = runtime_dir.to_string_lossy(); + let needs_reexec = std::env::var_os("DYLD_LIBRARY_PATH") + .is_none_or(|value| !value.to_string_lossy().contains(runtime_str.as_ref())); + if !needs_reexec { + return Ok(()); + } + + let mut dyld_paths = vec![runtime_dir.clone()]; + if let Some(existing) = std::env::var_os("DYLD_LIBRARY_PATH") { + dyld_paths.extend(std::env::split_paths(&existing)); + } + let joined = std::env::join_paths(&dyld_paths) + .map_err(|err| miette::miette!("join DYLD_LIBRARY_PATH: {err}"))?; + let exe = std::env::current_exe().into_diagnostic()?; + let args: Vec = std::env::args().skip(1).collect(); + + // Use execvp() so the current process is *replaced* by the re-exec'd + // binary — no wrapper process sits between the compute driver and + // the actually-running VM launcher. That avoids two problems: + // 1. An extra process level that survives SIGKILL of the driver + // (the wrapper was reparenting the re-exec'd child to init). + // 2. Signal forwarding: with a wrapper, a SIGTERM to the wrapper + // doesn't reach the child unless we hand-roll forwarding. + // After exec, the child inherits our PID and our procguard arming. + let err = std::process::Command::new(exe) + .args(&args) + .env("DYLD_LIBRARY_PATH", &joined) + .env(VM_RUNTIME_DIR_ENV, runtime_dir) + .env(REEXEC_ENV, "1") + .exec(); + // `exec()` only returns on failure. + Err(miette::miette!("failed to re-exec with runtime env: {err}")) +} + +#[cfg(not(target_os = "macos"))] +// Signature must match the macOS variant which can fail. +#[allow(clippy::unnecessary_wraps)] +fn maybe_reexec_internal_vm_with_runtime_env() -> Result<()> { + Ok(()) +} diff --git a/crates/openshell-driver-vm/src/procguard.rs b/crates/openshell-driver-vm/src/procguard.rs new file mode 100644 index 0000000000..fd4d3c872c --- /dev/null +++ b/crates/openshell-driver-vm/src/procguard.rs @@ -0,0 +1,198 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Cross-platform "die when my parent dies" primitive. +//! +//! The VM driver spawns a chain of subprocesses (compute driver → `--internal-run-vm` +//! launcher → gvproxy + libkrun fork). If any link in that chain is killed +//! with SIGKILL — or simply crashes — the children are reparented to init +//! and survive indefinitely, leaking libkrun workers and gvproxy +//! instances. +//! +//! This module exposes two functions: +//! * [`die_with_parent`] — configure the kernel (Linux) or a helper +//! thread (BSDs, incl. macOS) to SIGKILL the current process when its +//! parent dies. Call it from `main` in every subprocess we spawn +//! along the chain. Idempotent-ish (each call is a full setup — see +//! the runtime.rs comment at the single call site). +//! * [`die_with_parent_cleanup`] — same as above, but on the BSD path a +//! best-effort cleanup callback runs *before* this process exits. +//! This matters when we own a non-Rust child (e.g. gvproxy) that +//! cannot arm its own procguard; the callback lets us SIGTERM it +//! first. +//! +//! The Linux path uses `nix::sys::prctl::set_pdeathsig(SIGKILL)`, and +//! the BSD path uses `smol-rs/polling` with its `kqueue::Process` + +//! `ProcessOps::Exit` filter. Both are well-tested library surfaces; +//! we keep only the glue code and the pre-arming parent-liveness +//! re-check. + +/// Arrange for the current process to receive SIGKILL if its parent dies. +/// +/// On Linux this sets `PR_SET_PDEATHSIG` to SIGKILL (via +/// `nix::sys::prctl`). The kernel delivers SIGKILL the moment +/// `getppid()` changes away from the original parent. +/// +/// On the BSD family (macOS, FreeBSD, etc.) this spawns a detached +/// helper thread that uses `kqueue` with `EVFILT_PROC | NOTE_EXIT` on +/// the parent PID. When the parent exits the thread calls `exit(1)`, +/// which is sufficient for our use case — we are not a critical daemon +/// that needs to drain state; we are a VM launcher / gRPC driver whose +/// entire job is tied to the parent's lifetime. +pub fn die_with_parent() -> Result<(), String> { + die_with_parent_cleanup(|| ()) +} + +/// Like [`die_with_parent`], but run `cleanup` before terminating. +/// +/// The cleanup hook is best-effort and async-signal-unsafe — it runs on +/// the helper thread immediately before terminating the process. Use this +/// when we own children that cannot arm their own procguard; the cleanup +/// hook is the only chance we get to send them SIGTERM after the kernel +/// reparents us. +/// +/// On Linux the cleanup is a no-op: `PR_SET_PDEATHSIG` delivers SIGKILL +/// directly to us, there is no Rust-controlled moment between "parent +/// died" and "we die" in which we could run a callback. +pub fn die_with_parent_cleanup(cleanup: F) -> Result<(), String> +where + F: FnOnce() + Send + 'static, +{ + #[cfg(target_os = "linux")] + { + // Linux has no opportunity for a cleanup hook — the kernel + // delivers SIGKILL directly. Callers that need pre-exit cleanup + // must combine this with a `pre_exec` PR_SET_PDEATHSIG on their + // children (so the kernel cascades) or rely on process-group + // killpg from a signal handler in the parent. + let _ = cleanup; // intentionally dropped + install_linux_pdeathsig() + } + + #[cfg(any( + target_os = "macos", + target_os = "ios", + target_os = "freebsd", + target_os = "netbsd", + target_os = "openbsd", + target_os = "dragonfly", + ))] + { + install_bsd_kqueue_watcher(cleanup) + } + + #[cfg(not(any( + target_os = "linux", + target_os = "macos", + target_os = "ios", + target_os = "freebsd", + target_os = "netbsd", + target_os = "openbsd", + target_os = "dragonfly", + )))] + { + let _ = cleanup; + Ok(()) + } +} + +#[cfg(target_os = "linux")] +fn install_linux_pdeathsig() -> Result<(), String> { + use nix::sys::signal::Signal; + use nix::unistd::getppid; + + // Race: if the parent already died between fork/exec and this call, + // `getppid()` now returns 1 and PR_SET_PDEATHSIG will never fire. + // Read the current parent first so we can detect that case and exit. + let original_ppid = getppid(); + if original_ppid == nix::unistd::Pid::from_raw(1) { + return Err("process was already orphaned before procguard armed".to_string()); + } + + nix::sys::prctl::set_pdeathsig(Signal::SIGKILL) + .map_err(|err| format!("prctl(PR_SET_PDEATHSIG) failed: {err}"))?; + + // Re-check after arming: the parent may have died between getppid() + // and prctl(). If so, PR_SET_PDEATHSIG missed its window. + if getppid() != original_ppid { + return Err("parent exited before procguard could arm".to_string()); + } + + Ok(()) +} + +#[cfg(any( + target_os = "macos", + target_os = "ios", + target_os = "freebsd", + target_os = "netbsd", + target_os = "openbsd", + target_os = "dragonfly", +))] +fn install_bsd_kqueue_watcher(cleanup: F) -> Result<(), String> +where + F: FnOnce() + Send + 'static, +{ + use nix::unistd::getppid; + use polling::os::kqueue::{PollerKqueueExt, Process, ProcessOps}; + use polling::{Events, PollMode, Poller}; + + let parent_pid = getppid(); + if parent_pid == nix::unistd::Pid::from_raw(1) { + return Err("process was already orphaned before procguard armed".to_string()); + } + let parent_pid_nz = std::num::NonZeroI32::new(parent_pid.as_raw()) + .ok_or_else(|| "getppid returned 0 unexpectedly".to_string())?; + + // Build the poller on the caller's thread so any setup error + // surfaces synchronously. `EVFILT_PROC | NOTE_EXIT` is a one-shot + // filter, so `PollMode::Oneshot` matches the kernel semantics. + // + // SAFETY: `Process::from_pid` requires the PID to "be tied to an + // actual child process". Our parent is alive at this point — we + // re-check `getppid()` immediately after registration to close the + // race where the parent dies between the read above and the + // `add_filter` call. The BSD kqueue implementation accepts any + // live PID, not just our own children; the "child" wording in the + // polling docs is carried over from historical terminology in the + // kqueue(2) manpage. The kernel guarantees NOTE_EXIT fires if the + // PID is valid at registration. + let poller = Poller::new().map_err(|err| format!("polling: Poller::new failed: {err}"))?; + let key = 1; + #[allow(unsafe_code)] + // SAFETY requirement is documented on the enclosing function: the + // PID was just read from `getppid()` and re-checked below, so it + // points at a live process. `Process::from_pid` is an + // entry-in-the-kernel-table registration — the kernel validates + // the PID when the filter is added. + let filter = unsafe { Process::from_pid(parent_pid_nz, ProcessOps::Exit) }; + poller + .add_filter(filter, key, PollMode::Oneshot) + .map_err(|err| format!("polling: add_filter(NOTE_EXIT, {parent_pid_nz}) failed: {err}"))?; + + // Between getppid() and the registered filter the parent may + // already have died. Detect that and abort so the caller can bail. + if getppid() != parent_pid { + return Err("parent exited before procguard could arm".to_string()); + } + + // Hand off to a dedicated OS thread. Block in `poller.wait()` + // until the single NOTE_EXIT event fires, run the cleanup, then + // exit. We prefer `exit(1)` over `kill(getpid, SIGKILL)` so the + // callback gets to complete — SIGKILL would race it. Our children + // have their own procguards armed and will notice `getppid() == + // 1` shortly after, so we do not need Linux-semantics exactness. + std::thread::Builder::new() + .name("procguard".to_string()) + .spawn(move || { + let mut events = Events::new(); + // Block indefinitely; the filter is Oneshot so we expect + // exactly one event (parent's NOTE_EXIT) or a spurious + // wakeup we treat the same way. + let _ = poller.wait(&mut events, None); + let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(cleanup)); + std::process::exit(1); + }) + .map(|_| ()) + .map_err(|e| format!("failed to spawn procguard thread: {e}")) +} diff --git a/crates/openshell-driver-vm/src/rootfs.rs b/crates/openshell-driver-vm/src/rootfs.rs new file mode 100644 index 0000000000..5ea687d154 --- /dev/null +++ b/crates/openshell-driver-vm/src/rootfs.rs @@ -0,0 +1,452 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::fs; +use std::fs::File; +use std::io::{BufWriter, Cursor}; +use std::path::Path; + +const SUPERVISOR: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/openshell-sandbox.zst")); +const ROOTFS_VARIANT_MARKER: &str = ".openshell-rootfs-variant"; +const SANDBOX_GUEST_INIT_PATH: &str = "/srv/openshell-vm-sandbox-init.sh"; +const SANDBOX_SUPERVISOR_PATH: &str = "/opt/openshell/bin/openshell-sandbox"; + +pub const fn sandbox_guest_init_path() -> &'static str { + SANDBOX_GUEST_INIT_PATH +} + +pub fn prepare_sandbox_rootfs_from_image_root( + rootfs: &Path, + image_identity: &str, +) -> Result<(), String> { + prepare_sandbox_rootfs(rootfs)?; + validate_sandbox_rootfs(rootfs)?; + fs::write( + rootfs.join(ROOTFS_VARIANT_MARKER), + format!("{}:image:{image_identity}\n", env!("CARGO_PKG_VERSION")), + ) + .map_err(|e| format!("write rootfs variant marker: {e}"))?; + Ok(()) +} + +pub fn extract_rootfs_archive_to(archive_path: &Path, dest: &Path) -> Result<(), String> { + if dest.exists() { + fs::remove_dir_all(dest) + .map_err(|e| format!("remove old rootfs {}: {e}", dest.display()))?; + } + + fs::create_dir_all(dest).map_err(|e| format!("create rootfs dir {}: {e}", dest.display()))?; + let file = + File::open(archive_path).map_err(|e| format!("open {}: {e}", archive_path.display()))?; + let mut archive = tar::Archive::new(file); + archive + .unpack(dest) + .map_err(|e| format!("extract rootfs tarball into {}: {e}", dest.display())) +} + +pub fn create_rootfs_archive_from_dir(source: &Path, archive_path: &Path) -> Result<(), String> { + if let Some(parent) = archive_path.parent() { + fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?; + } + + let file = File::create(archive_path) + .map_err(|e| format!("create {}: {e}", archive_path.display()))?; + let writer = BufWriter::new(file); + let mut builder = tar::Builder::new(writer); + append_rootfs_tree_to_archive(&mut builder, source, Path::new("")).map_err(|e| { + format!( + "archive {} into {}: {e}", + source.display(), + archive_path.display() + ) + })?; + builder + .finish() + .map_err(|e| format!("finalize {}: {e}", archive_path.display())) +} + +fn append_rootfs_tree_to_archive( + builder: &mut tar::Builder>, + source: &Path, + archive_prefix: &Path, +) -> Result<(), String> { + let mut entries = fs::read_dir(source) + .map_err(|e| format!("read {}: {e}", source.display()))? + .collect::, _>>() + .map_err(|e| format!("read {}: {e}", source.display()))?; + entries.sort_by_key(fs::DirEntry::file_name); + + for entry in entries { + let entry_name = entry.file_name(); + let source_path = entry.path(); + let archive_path = if archive_prefix.as_os_str().is_empty() { + entry_name.into() + } else { + archive_prefix.join(entry_name) + }; + let metadata = fs::symlink_metadata(&source_path) + .map_err(|e| format!("stat {}: {e}", source_path.display()))?; + let file_type = metadata.file_type(); + + if file_type.is_dir() { + builder + .append_dir(&archive_path, &source_path) + .map_err(|e| format!("append dir {}: {e}", source_path.display()))?; + append_rootfs_tree_to_archive(builder, &source_path, &archive_path)?; + continue; + } + + if file_type.is_file() { + let mut file = File::open(&source_path) + .map_err(|e| format!("open {}: {e}", source_path.display()))?; + builder + .append_file(&archive_path, &mut file) + .map_err(|e| format!("append file {}: {e}", source_path.display()))?; + continue; + } + + if file_type.is_symlink() { + append_symlink_to_archive(builder, &source_path, &archive_path, &metadata)?; + continue; + } + + return Err(format!( + "unsupported rootfs entry type at {}", + source_path.display() + )); + } + + Ok(()) +} + +fn append_symlink_to_archive( + builder: &mut tar::Builder>, + source_path: &Path, + archive_path: &Path, + metadata: &fs::Metadata, +) -> Result<(), String> { + let target = fs::read_link(source_path) + .map_err(|e| format!("readlink {}: {e}", source_path.display()))?; + let mut header = tar::Header::new_gnu(); + header.set_metadata(metadata); + header.set_size(0); + header.set_cksum(); + builder + .append_link(&mut header, archive_path, target) + .map_err(|e| format!("append symlink {}: {e}", source_path.display())) +} + +fn prepare_sandbox_rootfs(rootfs: &Path) -> Result<(), String> { + for relative in [ + "usr/local/bin/k3s", + "usr/local/bin/kubectl", + "var/lib/rancher", + "etc/rancher", + "opt/openshell/charts", + "opt/openshell/manifests", + "opt/openshell/.initialized", + "opt/openshell/.rootfs-type", + ] { + remove_rootfs_path(rootfs, relative)?; + } + + let init_path = rootfs.join("srv/openshell-vm-sandbox-init.sh"); + if let Some(parent) = init_path.parent() { + fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?; + } + fs::write( + &init_path, + include_str!("../scripts/openshell-vm-sandbox-init.sh"), + ) + .map_err(|e| format!("write {}: {e}", init_path.display()))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + + fs::set_permissions(&init_path, fs::Permissions::from_mode(0o755)) + .map_err(|e| format!("chmod {}: {e}", init_path.display()))?; + } + + ensure_supervisor_binary(rootfs)?; + + let opt_dir = rootfs.join("opt/openshell"); + fs::create_dir_all(&opt_dir).map_err(|e| format!("create {}: {e}", opt_dir.display()))?; + fs::write(opt_dir.join(".rootfs-type"), "sandbox\n") + .map_err(|e| format!("write sandbox rootfs marker: {e}"))?; + ensure_sandbox_guest_user(rootfs)?; + + Ok(()) +} + +pub fn validate_sandbox_rootfs(rootfs: &Path) -> Result<(), String> { + require_rootfs_path(rootfs, SANDBOX_GUEST_INIT_PATH)?; + require_rootfs_path(rootfs, "/opt/openshell/bin/openshell-sandbox")?; + require_any_rootfs_path(rootfs, &["/bin/bash"])?; + require_any_rootfs_path(rootfs, &["/bin/mount", "/usr/bin/mount"])?; + require_any_rootfs_path( + rootfs, + &["/sbin/ip", "/usr/sbin/ip", "/bin/ip", "/usr/bin/ip"], + )?; + require_any_rootfs_path(rootfs, &["/bin/sed", "/usr/bin/sed"])?; + Ok(()) +} + +fn ensure_sandbox_guest_user(rootfs: &Path) -> Result<(), String> { + const SANDBOX_UID: u32 = 10001; + const SANDBOX_GID: u32 = 10001; + + let etc_dir = rootfs.join("etc"); + fs::create_dir_all(&etc_dir).map_err(|e| format!("create {}: {e}", etc_dir.display()))?; + + ensure_line_in_file( + &etc_dir.join("group"), + &format!("sandbox:x:{SANDBOX_GID}:"), + |line| line.starts_with("sandbox:"), + )?; + ensure_line_in_file(&etc_dir.join("gshadow"), "sandbox:!::", |line| { + line.starts_with("sandbox:") + })?; + ensure_line_in_file( + &etc_dir.join("passwd"), + &format!("sandbox:x:{SANDBOX_UID}:{SANDBOX_GID}:OpenShell Sandbox:/sandbox:/bin/bash"), + |line| line.starts_with("sandbox:"), + )?; + ensure_line_in_file( + &etc_dir.join("shadow"), + "sandbox:!:20123:0:99999:7:::", + |line| line.starts_with("sandbox:"), + )?; + + Ok(()) +} + +fn ensure_line_in_file( + path: &Path, + line: &str, + exists: impl Fn(&str) -> bool, +) -> Result<(), String> { + let mut contents = if path.exists() { + fs::read_to_string(path).map_err(|e| format!("read {}: {e}", path.display()))? + } else { + String::new() + }; + + if contents.lines().any(exists) { + return Ok(()); + } + + if !contents.is_empty() && !contents.ends_with('\n') { + contents.push('\n'); + } + contents.push_str(line); + contents.push('\n'); + + fs::write(path, contents).map_err(|e| format!("write {}: {e}", path.display())) +} + +fn ensure_supervisor_binary(rootfs: &Path) -> Result<(), String> { + let path = rootfs.join(SANDBOX_SUPERVISOR_PATH.trim_start_matches('/')); + if SUPERVISOR.is_empty() { + if !path.exists() { + return Err( + "sandbox supervisor not embedded. Build openshell-driver-vm with OPENSHELL_VM_RUNTIME_COMPRESSED_DIR set and run `mise run vm:setup && mise run vm:supervisor` first" + .to_string(), + ); + } + } else { + if let Some(parent) = path.parent() { + fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?; + } + + let supervisor = zstd::decode_all(Cursor::new(SUPERVISOR)) + .map_err(|e| format!("decompress supervisor: {e}"))?; + fs::write(&path, supervisor).map_err(|e| format!("write {}: {e}", path.display()))?; + } + + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + + fs::set_permissions(&path, fs::Permissions::from_mode(0o755)) + .map_err(|e| format!("chmod {}: {e}", path.display()))?; + } + + Ok(()) +} + +fn require_rootfs_path(rootfs: &Path, relative: &str) -> Result<(), String> { + let candidate = rootfs.join(relative.trim_start_matches('/')); + if candidate.exists() { + Ok(()) + } else { + Err(format!( + "prepared rootfs is missing {}", + candidate.display() + )) + } +} + +fn require_any_rootfs_path(rootfs: &Path, candidates: &[&str]) -> Result<(), String> { + if candidates + .iter() + .any(|candidate| rootfs.join(candidate.trim_start_matches('/')).exists()) + { + Ok(()) + } else { + Err(format!( + "prepared rootfs is missing one of: {}", + candidates.join(", ") + )) + } +} + +fn remove_rootfs_path(rootfs: &Path, relative: &str) -> Result<(), String> { + let path = rootfs.join(relative); + if !path.exists() { + return Ok(()); + } + + let result = if path.is_dir() { + fs::remove_dir_all(&path) + } else { + fs::remove_file(&path) + }; + result.map_err(|e| format!("remove {}: {e}", path.display())) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + use std::sync::atomic::{AtomicU64, Ordering}; + use std::time::{SystemTime, UNIX_EPOCH}; + + #[test] + fn prepare_sandbox_rootfs_rewrites_guest_layout() { + let dir = unique_temp_dir(); + let rootfs = dir.join("rootfs"); + + fs::create_dir_all(rootfs.join("usr/local/bin")).expect("create usr/local/bin"); + fs::create_dir_all(rootfs.join("etc")).expect("create etc"); + fs::create_dir_all(rootfs.join("var/lib/rancher")).expect("create var/lib/rancher"); + fs::create_dir_all(rootfs.join("opt/openshell/charts")).expect("create charts"); + fs::create_dir_all(rootfs.join("opt/openshell/manifests")).expect("create manifests"); + fs::create_dir_all(rootfs.join("opt/openshell/bin")).expect("create openshell bin"); + fs::write(rootfs.join("usr/local/bin/k3s"), b"k3s").expect("write k3s"); + fs::write(rootfs.join("usr/local/bin/kubectl"), b"kubectl").expect("write kubectl"); + fs::write(rootfs.join("opt/openshell/.initialized"), b"yes").expect("write initialized"); + fs::write( + rootfs.join("opt/openshell/bin/openshell-sandbox"), + b"sandbox", + ) + .expect("write openshell-sandbox"); + fs::write( + rootfs.join("etc/passwd"), + "root:x:0:0:root:/root:/bin/bash\n", + ) + .expect("write passwd"); + fs::write(rootfs.join("etc/group"), "root:x:0:\n").expect("write group"); + fs::write(rootfs.join("etc/hosts"), "127.0.0.1 localhost\n").expect("write hosts"); + fs::create_dir_all(rootfs.join("bin")).expect("create bin"); + fs::create_dir_all(rootfs.join("sbin")).expect("create sbin"); + fs::write(rootfs.join("bin/bash"), b"bash").expect("write bash"); + fs::write(rootfs.join("bin/mount"), b"mount").expect("write mount"); + fs::write(rootfs.join("bin/sed"), b"sed").expect("write sed"); + fs::write(rootfs.join("sbin/ip"), b"ip").expect("write ip"); + + prepare_sandbox_rootfs(&rootfs).expect("prepare sandbox rootfs"); + validate_sandbox_rootfs(&rootfs).expect("validate sandbox rootfs"); + + assert!(!rootfs.join("usr/local/bin/k3s").exists()); + assert!(!rootfs.join("usr/local/bin/kubectl").exists()); + assert!(!rootfs.join("var/lib/rancher").exists()); + assert!(!rootfs.join("opt/openshell/charts").exists()); + assert!(!rootfs.join("opt/openshell/manifests").exists()); + assert!(rootfs.join("srv/openshell-vm-sandbox-init.sh").is_file()); + assert!(!rootfs.join("sandbox").exists()); + assert!( + fs::read_to_string(rootfs.join("etc/passwd")) + .expect("read passwd") + .contains("sandbox:x:10001:10001:OpenShell Sandbox:/sandbox:/bin/bash") + ); + assert!( + fs::read_to_string(rootfs.join("etc/group")) + .expect("read group") + .contains("sandbox:x:10001:") + ); + assert_eq!( + fs::read_to_string(rootfs.join("etc/hosts")).expect("read hosts"), + "127.0.0.1 localhost\n" + ); + + let _ = fs::remove_dir_all(&dir); + } + + #[test] + fn prepare_sandbox_rootfs_preserves_image_workdir_contents() { + let dir = unique_temp_dir(); + let rootfs = dir.join("rootfs"); + + fs::create_dir_all(rootfs.join("opt/openshell/bin")).expect("create openshell bin"); + fs::write( + rootfs.join("opt/openshell/bin/openshell-sandbox"), + b"sandbox", + ) + .expect("write openshell-sandbox"); + fs::create_dir_all(rootfs.join("sandbox")).expect("create sandbox workdir"); + fs::write(rootfs.join("sandbox/app.py"), "print('hello')\n").expect("write app"); + + prepare_sandbox_rootfs(&rootfs).expect("prepare sandbox rootfs"); + + assert_eq!( + fs::read_to_string(rootfs.join("sandbox/app.py")).expect("read app"), + "print('hello')\n" + ); + + let _ = fs::remove_dir_all(&dir); + } + + #[cfg(unix)] + #[test] + fn create_rootfs_archive_preserves_broken_symlinks() { + let dir = unique_temp_dir(); + let rootfs = dir.join("rootfs"); + let extracted = dir.join("extracted"); + let archive = dir.join("rootfs.tar"); + + fs::create_dir_all(rootfs.join("etc")).expect("create etc"); + fs::write(rootfs.join("etc/hosts"), "127.0.0.1 localhost\n").expect("write hosts"); + std::os::unix::fs::symlink("/proc/self/mounts", rootfs.join("etc/mtab")) + .expect("create symlink"); + + create_rootfs_archive_from_dir(&rootfs, &archive).expect("archive rootfs"); + extract_rootfs_archive_to(&archive, &extracted).expect("extract rootfs"); + + let extracted_link = extracted.join("etc/mtab"); + assert!( + fs::symlink_metadata(&extracted_link) + .unwrap() + .file_type() + .is_symlink() + ); + assert_eq!( + fs::read_link(&extracted_link).expect("read extracted symlink"), + PathBuf::from("/proc/self/mounts") + ); + + let _ = fs::remove_dir_all(&dir); + } + + fn unique_temp_dir() -> PathBuf { + static COUNTER: AtomicU64 = AtomicU64::new(0); + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("time went backwards") + .as_nanos(); + let suffix = COUNTER.fetch_add(1, Ordering::Relaxed); + std::env::temp_dir().join(format!( + "openshell-driver-vm-rootfs-test-{}-{nanos}-{suffix}", + std::process::id() + )) + } +} diff --git a/crates/openshell-driver-vm/src/runtime.rs b/crates/openshell-driver-vm/src/runtime.rs new file mode 100644 index 0000000000..758808c8ee --- /dev/null +++ b/crates/openshell-driver-vm/src/runtime.rs @@ -0,0 +1,1387 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +#![allow(unsafe_code)] + +use std::ffi::CString; +use std::path::{Path, PathBuf}; +use std::process::{Child as StdChild, Command as StdCommand, Stdio}; +use std::ptr; +use std::sync::atomic::{AtomicI32, Ordering}; +use std::time::{Duration, Instant}; + +use crate::{embedded_runtime, ffi, procguard}; + +pub const VM_RUNTIME_DIR_ENV: &str = "OPENSHELL_VM_RUNTIME_DIR"; + +/// PID of the VM worker process (libkrun fork or QEMU). Zero when not running. +/// Used by the SIGTERM/SIGINT handler to forward signals to the VM. +static CHILD_PID: AtomicI32 = AtomicI32::new(0); + +/// PID of the helper process (gvproxy for libkrun, virtiofsd for QEMU). +/// Zero when not running. Used by the SIGTERM/SIGINT handler and +/// procguard cleanup callback to ensure the helper doesn't outlive the +/// launcher (especially on macOS where `PR_SET_PDEATHSIG` is absent). +static GVPROXY_PID: AtomicI32 = AtomicI32::new(0); + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum VmBackend { + Libkrun, + Qemu, +} + +// virtio-net feature bits (see Linux `include/uapi/linux/virtio_net.h`). +const NET_FEATURE_CSUM: u32 = 1 << 0; +const NET_FEATURE_GUEST_CSUM: u32 = 1 << 1; +const NET_FEATURE_GUEST_TSO4: u32 = 1 << 7; +const NET_FEATURE_GUEST_UFO: u32 = 1 << 10; +const NET_FEATURE_HOST_TSO4: u32 = 1 << 11; +const NET_FEATURE_HOST_UFO: u32 = 1 << 14; +const COMPAT_NET_FEATURES: u32 = NET_FEATURE_CSUM + | NET_FEATURE_GUEST_CSUM + | NET_FEATURE_GUEST_TSO4 + | NET_FEATURE_GUEST_UFO + | NET_FEATURE_HOST_TSO4 + | NET_FEATURE_HOST_UFO; + +pub struct VmLaunchConfig { + pub rootfs: PathBuf, + pub vcpus: u8, + pub mem_mib: u32, + pub exec_path: String, + pub args: Vec, + pub env: Vec, + pub workdir: String, + pub log_level: u32, + pub console_output: PathBuf, + pub backend: VmBackend, + pub gpu_bdf: Option, + pub tap_device: Option, + pub guest_ip: Option, + pub host_ip: Option, + pub vsock_cid: Option, + pub guest_mac: Option, + pub gateway_port: Option, +} + +pub fn run_vm(config: &VmLaunchConfig) -> Result<(), String> { + match config.backend { + VmBackend::Qemu => run_qemu_vm(config), + VmBackend::Libkrun => run_libkrun_vm(config), + } +} + +fn run_qemu_vm(config: &VmLaunchConfig) -> Result<(), String> { + let gpu_bdf = config + .gpu_bdf + .as_deref() + .ok_or("gpu_bdf is required for QEMU backend")?; + let tap_device = config + .tap_device + .as_deref() + .ok_or("tap_device is required for QEMU backend")?; + let guest_mac = config + .guest_mac + .as_deref() + .ok_or("guest_mac is required for QEMU backend")?; + let vsock_cid = config + .vsock_cid + .ok_or("vsock_cid is required for QEMU backend")?; + let _guest_ip = config + .guest_ip + .as_deref() + .ok_or("guest_ip is required for QEMU backend")?; + let host_ip = config + .host_ip + .as_deref() + .ok_or("host_ip is required for QEMU backend")?; + + if !config.rootfs.is_dir() { + return Err(format!( + "rootfs directory not found: {}", + config.rootfs.display() + )); + } + + if let Err(err) = procguard::die_with_parent_cleanup(procguard_kill_children) { + return Err(format!("procguard arm failed: {err}")); + } + + #[cfg(target_os = "linux")] + check_kvm_access()?; + + let guest_env = qemu_guest_env_vars(config, host_dns_server()); + write_guest_env_file(&config.rootfs, &guest_env)?; + + let rootfs_str = config.rootfs.to_str().ok_or("rootfs path not UTF-8")?; + let sandbox_dir = config.rootfs.parent().unwrap_or(&config.rootfs); + let sock_prefix = tap_device.trim_start_matches("vmtap-"); + let virtiofsd_sock_dir = PathBuf::from(format!("/tmp/ovm-qemu-{sock_prefix}")); + std::fs::create_dir_all(&virtiofsd_sock_dir) + .map_err(|e| format!("create virtiofsd sock dir: {e}"))?; + let virtiofsd_sock = virtiofsd_sock_dir.join("virtiofsd.sock"); + let shm_path = format!("/dev/shm/ovm-qemu-{sock_prefix}"); + + std::fs::create_dir_all(&shm_path).map_err(|e| format!("create shm dir: {e}"))?; + + let runtime_dir = qemu_runtime_dir()?; + + let gw_port = config.gateway_port.unwrap_or(0); + setup_tap_networking(tap_device, host_ip, gw_port)?; + let mut tap_guard = TapGuard::new(tap_device.to_string(), host_ip.to_string(), gw_port); + + let virtiofsd_log = sandbox_dir.join("virtiofsd.log"); + let virtiofsd_log_file = + std::fs::File::create(&virtiofsd_log).map_err(|e| format!("create virtiofsd log: {e}"))?; + + let virtiofsd_bin = { + let runtime_virtiofsd = runtime_dir.join("virtiofsd"); + if runtime_virtiofsd.is_file() { + runtime_virtiofsd + } else { + PathBuf::from("virtiofsd") + } + }; + + let mut virtiofsd_cmd = StdCommand::new(&virtiofsd_bin); + virtiofsd_cmd + .arg("--socket-path") + .arg(&virtiofsd_sock) + .arg("--shared-dir") + .arg(rootfs_str) + .arg("--cache=auto") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(virtiofsd_log_file); + + #[cfg(target_os = "linux")] + { + use nix::sys::signal::Signal; + use std::os::unix::process::CommandExt as _; + unsafe { + virtiofsd_cmd.pre_exec(|| { + nix::sys::prctl::set_pdeathsig(Signal::SIGKILL) + .map_err(|err| std::io::Error::other(format!("pdeathsig: {err}"))) + }); + } + } + + let virtiofsd_child = virtiofsd_cmd + .spawn() + .map_err(|e| format!("failed to start virtiofsd: {e}"))?; + let virtiofsd_pid = virtiofsd_child.id().cast_signed(); + GVPROXY_PID.store(virtiofsd_pid, Ordering::Relaxed); + let mut virtiofsd_guard = GvproxyGuard::new(virtiofsd_child); + + wait_for_path(&virtiofsd_sock, Duration::from_secs(5), "virtiofsd socket")?; + + let vmlinux = runtime_dir.join("vmlinux"); + if !vmlinux.is_file() { + return Err(format!("VM kernel not found: {}", vmlinux.display())); + } + + let kernel_cmdline = build_kernel_cmdline(config); + + let mut qemu_cmd = StdCommand::new("qemu-system-x86_64"); + qemu_cmd + .arg("-machine") + .arg("q35,accel=kvm") + .arg("-cpu") + .arg("host") + .arg("-smp") + .arg(config.vcpus.to_string()) + .arg("-m") + .arg(format!("{}M", config.mem_mib)) + .arg("-nographic") + .arg("-no-reboot") + .arg("-kernel") + .arg(&vmlinux) + .arg("-append") + .arg(&kernel_cmdline) + .arg("-chardev") + .arg(format!( + "socket,id=virtiofs,path={}", + virtiofsd_sock.display() + )) + .arg("-device") + .arg("vhost-user-fs-pci,chardev=virtiofs,tag=rootfs") + .arg("-object") + .arg(format!( + "memory-backend-memfd,id=mem,size={}M,share=on", + config.mem_mib + )) + .arg("-numa") + .arg("node,memdev=mem") + .arg("-netdev") + .arg(format!( + "tap,id=net0,ifname={tap_device},script=no,downscript=no" + )) + .arg("-device") + .arg("pcie-root-port,id=net_root,slot=3") + .arg("-device") + .arg(format!( + "virtio-net-pci-non-transitional,netdev=net0,mac={guest_mac},bus=net_root" + )) + .arg("-device") + .arg("pcie-root-port,id=vsock_root,slot=1") + .arg("-device") + .arg(format!( + "vhost-vsock-pci,guest-cid={vsock_cid},bus=vsock_root" + )) + .arg("-device") + .arg("pcie-root-port,id=gpu_root,slot=2") + .arg("-device") + .arg(format!("vfio-pci,host={gpu_bdf},bus=gpu_root")) + .arg("-serial") + .arg(format!("file:{}", config.console_output.display())); + + qemu_cmd.stdin(Stdio::null()); + qemu_cmd.stdout(Stdio::inherit()); + qemu_cmd.stderr(Stdio::inherit()); + + #[cfg(target_os = "linux")] + { + use nix::sys::signal::Signal; + use std::os::unix::process::CommandExt as _; + unsafe { + qemu_cmd.pre_exec(|| { + nix::sys::prctl::set_pdeathsig(Signal::SIGKILL) + .map_err(|err| std::io::Error::other(format!("pdeathsig: {err}"))) + }); + } + } + + let mut qemu_child = qemu_cmd + .spawn() + .map_err(|e| format!("failed to start QEMU: {e}"))?; + + let qemu_pid = qemu_child.id().cast_signed(); + install_signal_forwarding(qemu_pid); + + let status = qemu_child + .wait() + .map_err(|e| format!("failed to wait for QEMU: {e}"))?; + + CHILD_PID.store(0, Ordering::Relaxed); + unsafe { + libc::kill(virtiofsd_pid, libc::SIGTERM); + } + virtiofsd_guard.disarm(); + GVPROXY_PID.store(0, Ordering::Relaxed); + teardown_tap_networking(tap_device, host_ip, gw_port); + tap_guard.disarm(); + let _ = std::fs::remove_dir_all(&shm_path); + let _ = std::fs::remove_dir_all(&virtiofsd_sock_dir); + + if status.success() { + Ok(()) + } else { + Err(format!("QEMU exited with status {status}")) + } +} + +/// Write environment variables into the rootfs so the guest init script +/// can source them. virtiofs shares the host rootfs directory into the guest. +fn write_guest_env_file(rootfs: &Path, env_vars: &[String]) -> Result<(), String> { + let srv_dir = rootfs.join("srv"); + std::fs::create_dir_all(&srv_dir).map_err(|e| format!("create /srv in rootfs: {e}"))?; + let env_file = srv_dir.join("openshell-env.sh"); + let mut content = String::new(); + for var in env_vars { + if let Some((key, value)) = var.split_once('=') { + use std::fmt::Write as _; + let _ = writeln!(content, "export {key}=\"{}\"", shell_escape(value)); + } + } + std::fs::write(&env_file, &content).map_err(|e| format!("write guest env file: {e}"))?; + Ok(()) +} + +fn qemu_guest_env_vars(config: &VmLaunchConfig, dns_server: Option) -> Vec { + let mut env_vars = config.env.clone(); + + if let Some(ip) = &config.guest_ip + && let Some(host_ip) = &config.host_ip + { + env_vars.push(format!("VM_NET_IP={ip}")); + env_vars.push(format!("VM_NET_GW={host_ip}")); + } + + if let Some(dns) = dns_server { + env_vars.push(format!("VM_NET_DNS={dns}")); + } + + if config.gpu_bdf.is_some() { + env_vars.push("GPU_ENABLED=true".to_string()); + } + + env_vars +} + +/// Escape a string for use inside bash double quotes. +fn shell_escape(s: &str) -> String { + s.replace('\\', "\\\\") + .replace('"', "\\\"") + .replace('$', "\\$") + .replace('`', "\\`") + .replace('\n', "\\n") + .replace('\r', "\\r") +} + +fn build_kernel_cmdline(config: &VmLaunchConfig) -> String { + let mut parts = vec![ + "console=ttyS0".to_string(), + "root=rootfs".to_string(), + "rootfstype=virtiofs".to_string(), + "rw".to_string(), + "panic=-1".to_string(), + format!("init={}", config.exec_path), + ]; + + if let Some(ip) = &config.guest_ip + && let Some(host_ip) = &config.host_ip + { + parts.push(format!("ip={ip}::{host_ip}:255.255.255.252:sandbox::off")); + } + + if config.gpu_bdf.is_some() { + parts.push("firmware_class.path=/lib/firmware".to_string()); + } + + parts.join(" ") +} + +fn host_dns_server() -> Option { + // Prefer systemd-resolved upstream config (skips the 127.0.0.53 + // stub listener which is unreachable from inside QEMU/TAP guests). + for path in &["/run/systemd/resolve/resolv.conf", "/etc/resolv.conf"] { + let Ok(resolv) = std::fs::read_to_string(path) else { + continue; + }; + for line in resolv.lines() { + let line = line.trim(); + if let Some(server) = line.strip_prefix("nameserver") { + let server = server.trim(); + if server == "127.0.0.53" || server.starts_with("127.") { + continue; + } + if !server.is_empty() { + return Some(server.to_string()); + } + } + } + } + None +} + +/// Remove leftover `vmtap-*` interfaces from previous driver runs. +/// +/// Called once at driver startup for interfaces that were not torn down +/// (e.g. the launcher was `SIGKILL`-ed before teardown), so stale +/// interfaces cannot cause subnet routing conflicts with newly allocated TAPs. +pub fn cleanup_stale_tap_interfaces() { + let Ok(entries) = std::fs::read_dir("/sys/class/net") else { + return; + }; + for entry in entries.flatten() { + let name = entry.file_name(); + let Some(name) = name.to_str() else { + continue; + }; + if !name.starts_with("vmtap-") { + continue; + } + // Read the IP address so we can clean up iptables rules too. + // Port 0 tells teardown we don't know the original gateway port; + // the blanket legacy rule is still cleaned up best-effort. + let ip = read_tap_host_ip(name); + if let Some(ref host_ip) = ip { + teardown_tap_networking(name, host_ip, 0); + } else { + let _ = run_cmd("ip", &["link", "set", name, "down"]); + let _ = run_cmd("ip", &["tuntap", "del", "dev", name, "mode", "tap"]); + } + tracing::warn!(interface = %name, "removed stale TAP interface from previous run"); + } +} + +/// Read the first IPv4 address assigned to a network interface. +fn read_tap_host_ip(device: &str) -> Option { + let output = StdCommand::new("ip") + .args(["-4", "-o", "addr", "show", "dev", device]) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .output() + .ok()?; + let stdout = String::from_utf8_lossy(&output.stdout); + // Format: "28: vmtap-xxx inet 10.0.128.1/30 ..." + for token in stdout.split_whitespace() { + if let Some((ip, _prefix)) = token.split_once('/') + && ip.parse::().is_ok() + { + return Some(ip.to_string()); + } + } + None +} + +fn setup_tap_networking(tap_device: &str, host_ip: &str, gateway_port: u16) -> Result<(), String> { + run_cmd("ip", &["tuntap", "add", "dev", tap_device, "mode", "tap"])?; + run_cmd( + "ip", + &["addr", "add", &format!("{host_ip}/30"), "dev", tap_device], + )?; + run_cmd("ip", &["link", "set", tap_device, "up"])?; + + // Deprioritize routes through down interfaces so a stale vmtap-* + // that somehow survives cleanup cannot shadow the active one. + let _ = std::fs::write( + format!("/proc/sys/net/ipv4/conf/{tap_device}/ignore_routes_with_linkdown"), + "1", + ); + + enable_ip_forwarding()?; + + let subnet = tap_subnet_from_host_ip(host_ip); + let _ = run_cmd( + "iptables", + &[ + "-t", + "nat", + "-D", + "POSTROUTING", + "-s", + &subnet, + "-j", + "MASQUERADE", + ], + ); + run_cmd( + "iptables", + &[ + "-t", + "nat", + "-A", + "POSTROUTING", + "-s", + &subnet, + "-j", + "MASQUERADE", + ], + )?; + let _ = run_cmd( + "iptables", + &["-D", "FORWARD", "-i", tap_device, "-j", "ACCEPT"], + ); + run_cmd( + "iptables", + &["-A", "FORWARD", "-i", tap_device, "-j", "ACCEPT"], + )?; + let _ = run_cmd( + "iptables", + &[ + "-D", + "FORWARD", + "-o", + tap_device, + "-m", + "state", + "--state", + "RELATED,ESTABLISHED", + "-j", + "ACCEPT", + ], + ); + run_cmd( + "iptables", + &[ + "-A", + "FORWARD", + "-o", + tap_device, + "-m", + "state", + "--state", + "RELATED,ESTABLISHED", + "-j", + "ACCEPT", + ], + )?; + // Allow guest → host traffic only to the gateway gRPC port. + // Previous versions accepted ALL inbound traffic from the TAP + // interface; scope to the specific port so the guest cannot reach + // other host services. + let port_str = gateway_port.to_string(); + let _ = run_cmd( + "iptables", + &[ + "-D", "INPUT", "-i", tap_device, "-p", "tcp", "--dport", &port_str, "-j", "ACCEPT", + ], + ); + run_cmd( + "iptables", + &[ + "-A", "INPUT", "-i", tap_device, "-p", "tcp", "--dport", &port_str, "-j", "ACCEPT", + ], + )?; + + Ok(()) +} + +fn teardown_tap_networking(tap_device: &str, host_ip: &str, gateway_port: u16) { + let subnet = tap_subnet_from_host_ip(host_ip); + let _ = run_cmd( + "iptables", + &[ + "-D", + "FORWARD", + "-o", + tap_device, + "-m", + "state", + "--state", + "RELATED,ESTABLISHED", + "-j", + "ACCEPT", + ], + ); + let _ = run_cmd( + "iptables", + &["-D", "FORWARD", "-i", tap_device, "-j", "ACCEPT"], + ); + // Remove the port-scoped INPUT rule. Also try the legacy blanket + // rule so stale rules from older driver versions are cleaned up. + if gateway_port > 0 { + let port_str = gateway_port.to_string(); + let _ = run_cmd( + "iptables", + &[ + "-D", "INPUT", "-i", tap_device, "-p", "tcp", "--dport", &port_str, "-j", "ACCEPT", + ], + ); + } + let _ = run_cmd( + "iptables", + &["-D", "INPUT", "-i", tap_device, "-j", "ACCEPT"], + ); + let _ = run_cmd( + "iptables", + &[ + "-t", + "nat", + "-D", + "POSTROUTING", + "-s", + &subnet, + "-j", + "MASQUERADE", + ], + ); + let _ = run_cmd("ip", &["link", "set", tap_device, "down"]); + let _ = run_cmd("ip", &["tuntap", "del", "dev", tap_device, "mode", "tap"]); +} + +fn tap_subnet_from_host_ip(host_ip: &str) -> String { + host_ip.parse::().map_or_else( + |_| format!("{host_ip}/30"), + |ip| { + let base = u32::from(ip) & !3; + let base_ip = std::net::Ipv4Addr::from(base); + format!("{base_ip}/30") + }, + ) +} + +fn enable_ip_forwarding() -> Result<(), String> { + std::fs::write("/proc/sys/net/ipv4/ip_forward", "1") + .map_err(|e| format!("enable ip_forward: {e}")) +} + +fn run_cmd(cmd: &str, args: &[&str]) -> Result<(), String> { + let output = StdCommand::new(cmd) + .args(args) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .map_err(|e| format!("failed to run {cmd}: {e}"))?; + if output.status.success() { + Ok(()) + } else { + let stderr = String::from_utf8_lossy(&output.stderr); + Err(format!("{cmd} {} failed: {stderr}", args.join(" "))) + } +} + +/// RAII guard that tears down TAP networking on drop. +struct TapGuard { + tap_device: String, + host_ip: String, + gateway_port: u16, + disarmed: bool, +} + +impl TapGuard { + fn new(tap_device: String, host_ip: String, gateway_port: u16) -> Self { + Self { + tap_device, + host_ip, + gateway_port, + disarmed: false, + } + } + + fn disarm(&mut self) { + self.disarmed = true; + } +} + +impl Drop for TapGuard { + fn drop(&mut self) { + if !self.disarmed { + teardown_tap_networking(&self.tap_device, &self.host_ip, self.gateway_port); + } + } +} + +/// Shared procguard cleanup callback for both libkrun and QEMU paths. +/// Only async-signal-safe calls: atomic loads and `kill(2)`. +fn procguard_kill_children() { + let helper_pid = GVPROXY_PID.load(Ordering::Relaxed); + let child_pid = CHILD_PID.load(Ordering::Relaxed); + if helper_pid > 0 { + unsafe { + libc::kill(helper_pid, libc::SIGTERM); + } + } + if child_pid > 0 { + unsafe { + libc::kill(child_pid, libc::SIGTERM); + } + } + std::thread::sleep(Duration::from_millis(200)); + if helper_pid > 0 { + unsafe { + libc::kill(helper_pid, libc::SIGKILL); + } + } + if child_pid > 0 { + unsafe { + libc::kill(child_pid, libc::SIGKILL); + } + } +} + +fn run_libkrun_vm(config: &VmLaunchConfig) -> Result<(), String> { + if !config.rootfs.is_dir() { + return Err(format!( + "rootfs directory not found: {}", + config.rootfs.display() + )); + } + + // Arm procguard first, BEFORE we spawn gvproxy or fork libkrun, so + // that the launcher can't be orphaned during setup. The cleanup + // callback reads the GVPROXY_PID atomic (initially 0 — no-op) and + // the CHILD_PID atomic (the libkrun fork), so it stays correct as + // those slots get populated later in this function. Only ONE arm + // per process: racing two watchers for the same NOTE_EXIT event + // would cause whichever wins to skip the cleanup. + if let Err(err) = procguard::die_with_parent_cleanup(procguard_kill_children) { + return Err(format!("procguard arm failed: {err}")); + } + + #[cfg(target_os = "linux")] + check_kvm_access()?; + + let runtime_dir = configured_runtime_dir()?; + validate_runtime_dir(&runtime_dir)?; + configure_runtime_loader_env(&runtime_dir)?; + raise_nofile_limit(); + + let vm = VmContext::create(&runtime_dir, config.log_level)?; + vm.set_vm_config(config.vcpus, config.mem_mib)?; + vm.set_root(&config.rootfs)?; + vm.set_workdir(&config.workdir)?; + + // Run gvproxy strictly as the guest's virtual NIC / DHCP / router. + // + // After the supervisor-initiated relay migration (#867), the driver + // no longer forwards any host-side ports into the guest — all ingress + // traffic for SSH and exec rides the outbound `ConnectSupervisor` + // gRPC stream the guest opens to the gateway. What gvproxy still + // provides here is the TCP/IP *plane* the guest kernel needs: + // + // * a virtio-net backend attached to libkrun via a Unix + // SOCK_STREAM (Linux) or SOCK_DGRAM (macOS vfkit), which + // surfaces as `eth0` inside the guest; + // * the DHCP server + default router the guest's udhcpc client + // talks to on boot (IPs 192.168.127.1 / .2, defaults for + // gvisor-tap-vsock); + // * the host-facing gateway identity the guest uses for callbacks: + // gvproxy installs a default NAT entry rewriting `192.168.127.254` + // (the subnet's HostIP) to the host's `127.0.0.1`, and serves + // `host.containers.internal` / `host.docker.internal` / + // `host.openshell.internal` in its embedded DNS pointing at that + // same HostIP. The guest init script seeds /etc/hosts with the + // same mapping so the supervisor reaches the host gateway even + // when gvproxy's DNS isn't in resolv.conf. The gateway IP + // (192.168.127.1) is NOT a host-loopback proxy — it only listens + // on its own service ports (DNS:53, DHCP, HTTP API:80). + // + // That network plane is also what the sandbox supervisor's + // per-sandbox netns (veth pair + iptables, see + // `openshell-sandbox/src/sandbox/linux/netns.rs`) branches off of; + // libkrun's built-in TSI socket impersonation would not satisfy + // those kernel-level primitives. + // + // The `-listen` API socket and `-ssh-port` forwarder are both + // deliberately omitted: nothing in the driver enqueues port + // forwards on the API any more, and the host-side SSH listener is + // dead plumbing. + let gvproxy_guard = { + let gvproxy_binary = runtime_dir.join("gvproxy"); + if !gvproxy_binary.is_file() { + return Err(format!( + "missing runtime file: {}", + gvproxy_binary.display() + )); + } + + let sock_base = gvproxy_socket_base(&config.rootfs)?; + let net_sock = sock_base.with_extension("v"); + let _ = std::fs::remove_file(&net_sock); + let _ = std::fs::remove_file(sock_base.with_extension("v-krun.sock")); + + let run_dir = config.rootfs.parent().unwrap_or(&config.rootfs); + let gvproxy_log = run_dir.join("gvproxy.log"); + let gvproxy_log_file = std::fs::File::create(&gvproxy_log) + .map_err(|e| format!("create gvproxy log {}: {e}", gvproxy_log.display()))?; + + #[cfg(target_os = "linux")] + let (gvproxy_net_flag, gvproxy_net_url) = + ("-listen-qemu", format!("unix://{}", net_sock.display())); + #[cfg(target_os = "macos")] + let (gvproxy_net_flag, gvproxy_net_url) = ( + "-listen-vfkit", + format!("unixgram://{}", net_sock.display()), + ); + + // `-ssh-port -1` tells gvproxy to skip its default SSH forward + // (127.0.0.1:2222 → guest:22). We don't use it — all gateway + // ingress rides the supervisor-initiated relay — and leaving + // the default on would bind a host-side TCP listener per + // sandbox, racing concurrent sandboxes for port 2222 and + // surfacing a misleading "sshd is reachable" endpoint. See + // https://github.com/containers/gvisor-tap-vsock `cmd/gvproxy/main.go` + // (`getForwardsMap` returns an empty map when `sshPort == -1`). + let mut gvproxy_cmd = StdCommand::new(&gvproxy_binary); + gvproxy_cmd + .arg(gvproxy_net_flag) + .arg(&gvproxy_net_url) + .arg("-ssh-port") + .arg("-1") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(gvproxy_log_file); + + // On Linux the kernel will SIGKILL gvproxy the moment this + // launcher dies (or is SIGKILLed). `pre_exec` runs in the child + // between fork and execve, so the PR_SET_PDEATHSIG flag is + // inherited across execve and applies to gvproxy proper. On + // macOS/BSDs there is no equivalent; we fall back to killing + // gvproxy explicitly from the launcher's procguard cleanup + // callback (see `run_vm` above) and SIGTERM handler + // (see `install_signal_forwarding` below). + #[cfg(target_os = "linux")] + { + use nix::sys::signal::Signal; + use std::os::unix::process::CommandExt as _; + unsafe { + gvproxy_cmd.pre_exec(|| { + nix::sys::prctl::set_pdeathsig(Signal::SIGKILL) + .map_err(|err| std::io::Error::other(format!("pdeathsig: {err}"))) + }); + } + } + + let child = gvproxy_cmd + .spawn() + .map_err(|e| format!("failed to start gvproxy {}: {e}", gvproxy_binary.display()))?; + // The procguard cleanup reads GVPROXY_PID atomically. Storing it + // here makes the callback able to SIGTERM gvproxy if the driver + // dies from this moment onward. + GVPROXY_PID.store(child.id().cast_signed(), Ordering::Relaxed); + + wait_for_path(&net_sock, Duration::from_secs(5), "gvproxy data socket")?; + + vm.disable_implicit_vsock()?; + vm.add_vsock(0)?; + + let mac: [u8; 6] = [0x5a, 0x94, 0xef, 0xe4, 0x0c, 0xee]; + + #[cfg(target_os = "linux")] + vm.add_net_unixstream(&net_sock, &mac, COMPAT_NET_FEATURES)?; + #[cfg(target_os = "macos")] + { + const NET_FLAG_VFKIT: u32 = 1 << 0; + vm.add_net_unixgram(&net_sock, &mac, COMPAT_NET_FEATURES, NET_FLAG_VFKIT)?; + } + + Some(GvproxyGuard::new(child)) + }; + + vm.set_console_output(&config.console_output)?; + + let env = if config.env.is_empty() { + vec![ + "HOME=/root".to_string(), + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin".to_string(), + "TERM=xterm".to_string(), + ] + } else { + config.env.clone() + }; + vm.set_exec(&config.exec_path, &config.args, &env)?; + + let pid = unsafe { libc::fork() }; + match pid { + -1 => Err(format!("fork failed: {}", std::io::Error::last_os_error())), + 0 => { + // We are the libkrun worker (the VM's PID 1 inside the guest + // kernel, but a normal host process until krun_start_enter + // fires). Arm procguard so this fork is SIGKILLed if the + // parent launcher dies abruptly. On Linux this uses + // `PR_SET_PDEATHSIG`; on macOS this spawns a kqueue + // NOTE_EXIT watcher thread. Either way it closes the same + // leak gvproxy does above. + // + // We also SIGKILL ourselves if arming fails — there's no + // safe way to continue if we can't guarantee cleanup. + if let Err(err) = procguard::die_with_parent() { + eprintln!("libkrun worker: procguard arm failed: {err}"); + std::process::exit(1); + } + let ret = vm.start_enter(); + eprintln!("krun_start_enter failed: {ret}"); + std::process::exit(1); + } + _ => { + install_signal_forwarding(pid); + + let status = wait_for_child(pid)?; + CHILD_PID.store(0, Ordering::Relaxed); + cleanup_gvproxy(gvproxy_guard); + GVPROXY_PID.store(0, Ordering::Relaxed); + + if libc::WIFEXITED(status) { + match libc::WEXITSTATUS(status) { + 0 => Ok(()), + code => Err(format!("VM exited with status {code}")), + } + } else if libc::WIFSIGNALED(status) { + let sig = libc::WTERMSIG(status); + Err(format!("VM killed by signal {sig}")) + } else { + Err(format!("VM exited with unexpected wait status {status}")) + } + } + } +} + +pub fn validate_runtime_dir(dir: &Path) -> Result<(), String> { + if !dir.is_dir() { + return Err(format!( + "VM runtime not found at {}. Run `mise run vm:setup` or set {VM_RUNTIME_DIR_ENV}", + dir.display() + )); + } + + embedded_runtime::validate_runtime_dir(dir) +} + +pub fn configured_runtime_dir() -> Result { + if let Some(path) = std::env::var_os(VM_RUNTIME_DIR_ENV) { + return Ok(PathBuf::from(path)); + } + embedded_runtime::ensure_runtime_extracted() +} + +fn qemu_runtime_dir() -> Result { + configured_runtime_dir().map_err(|_| { + "QEMU backend requires OPENSHELL_VM_RUNTIME_DIR to be set (pointing to a directory \ + containing vmlinux). Set the env var or run `mise run vm:setup`." + .to_string() + }) +} + +#[cfg(target_os = "macos")] +fn configure_runtime_loader_env(runtime_dir: &Path) -> Result<(), String> { + let existing = std::env::var_os("DYLD_FALLBACK_LIBRARY_PATH"); + let mut paths = vec![runtime_dir.to_path_buf()]; + if let Some(existing) = existing { + paths.extend(std::env::split_paths(&existing)); + } + let joined = + std::env::join_paths(paths).map_err(|e| format!("join DYLD_FALLBACK_LIBRARY_PATH: {e}"))?; + unsafe { + std::env::set_var("DYLD_FALLBACK_LIBRARY_PATH", joined); + } + Ok(()) +} + +#[cfg(target_os = "linux")] +fn configure_runtime_loader_env(runtime_dir: &Path) -> Result<(), String> { + let existing = std::env::var_os("LD_LIBRARY_PATH"); + let mut paths = vec![runtime_dir.to_path_buf()]; + if let Some(existing) = existing { + paths.extend(std::env::split_paths(&existing)); + } + let joined = std::env::join_paths(paths).map_err(|e| format!("join LD_LIBRARY_PATH: {e}"))?; + unsafe { + std::env::set_var("LD_LIBRARY_PATH", joined); + } + Ok(()) +} + +#[cfg(not(any(target_os = "macos", target_os = "linux")))] +fn configure_runtime_loader_env(_runtime_dir: &Path) -> Result<(), String> { + Ok(()) +} + +fn raise_nofile_limit() { + #[cfg(unix)] + unsafe { + let mut rlim = libc::rlimit { + rlim_cur: 0, + rlim_max: 0, + }; + if libc::getrlimit(libc::RLIMIT_NOFILE, &raw mut rlim) == 0 { + rlim.rlim_cur = rlim.rlim_max; + let _ = libc::setrlimit(libc::RLIMIT_NOFILE, &raw const rlim); + } + } +} + +fn clamp_log_level(level: u32) -> u32 { + match level { + 0 => ffi::KRUN_LOG_LEVEL_OFF, + 1 => ffi::KRUN_LOG_LEVEL_ERROR, + 2 => ffi::KRUN_LOG_LEVEL_WARN, + 3 => ffi::KRUN_LOG_LEVEL_INFO, + 4 => ffi::KRUN_LOG_LEVEL_DEBUG, + _ => ffi::KRUN_LOG_LEVEL_TRACE, + } +} + +struct VmContext { + krun: &'static ffi::LibKrun, + ctx_id: u32, +} + +impl VmContext { + fn create(runtime_dir: &Path, log_level: u32) -> Result { + let krun = ffi::libkrun(runtime_dir)?; + check( + unsafe { + (krun.krun_init_log)( + ffi::KRUN_LOG_TARGET_DEFAULT, + clamp_log_level(log_level), + ffi::KRUN_LOG_STYLE_AUTO, + ffi::KRUN_LOG_OPTION_NO_ENV, + ) + }, + "krun_init_log", + )?; + + let ctx_id = unsafe { (krun.krun_create_ctx)() }; + if ctx_id < 0 { + return Err(format!("krun_create_ctx failed with error code {ctx_id}")); + } + + Ok(Self { + krun, + ctx_id: ctx_id.cast_unsigned(), + }) + } + + fn set_vm_config(&self, vcpus: u8, mem_mib: u32) -> Result<(), String> { + check( + unsafe { (self.krun.krun_set_vm_config)(self.ctx_id, vcpus, mem_mib) }, + "krun_set_vm_config", + ) + } + + fn set_root(&self, rootfs: &Path) -> Result<(), String> { + let rootfs_c = path_to_cstring(rootfs)?; + check( + unsafe { (self.krun.krun_set_root)(self.ctx_id, rootfs_c.as_ptr()) }, + "krun_set_root", + ) + } + + fn set_workdir(&self, workdir: &str) -> Result<(), String> { + let workdir_c = CString::new(workdir).map_err(|e| format!("invalid workdir: {e}"))?; + check( + unsafe { (self.krun.krun_set_workdir)(self.ctx_id, workdir_c.as_ptr()) }, + "krun_set_workdir", + ) + } + + fn disable_implicit_vsock(&self) -> Result<(), String> { + check( + unsafe { (self.krun.krun_disable_implicit_vsock)(self.ctx_id) }, + "krun_disable_implicit_vsock", + ) + } + + fn add_vsock(&self, tsi_features: u32) -> Result<(), String> { + check( + unsafe { (self.krun.krun_add_vsock)(self.ctx_id, tsi_features) }, + "krun_add_vsock", + ) + } + + #[cfg(target_os = "macos")] + fn add_net_unixgram( + &self, + socket_path: &Path, + mac: &[u8; 6], + features: u32, + flags: u32, + ) -> Result<(), String> { + let sock_c = path_to_cstring(socket_path)?; + check( + unsafe { + (self.krun.krun_add_net_unixgram)( + self.ctx_id, + sock_c.as_ptr(), + -1, + mac.as_ptr(), + features, + flags, + ) + }, + "krun_add_net_unixgram", + ) + } + + #[allow(dead_code)] // Used on Linux when gvproxy runs in qemu/unixstream mode. + fn add_net_unixstream( + &self, + socket_path: &Path, + mac: &[u8; 6], + features: u32, + ) -> Result<(), String> { + let sock_c = path_to_cstring(socket_path)?; + check( + unsafe { + (self.krun.krun_add_net_unixstream)( + self.ctx_id, + sock_c.as_ptr(), + -1, + mac.as_ptr(), + features, + 0, + ) + }, + "krun_add_net_unixstream", + ) + } + + fn set_console_output(&self, path: &Path) -> Result<(), String> { + let console_c = path_to_cstring(path)?; + check( + unsafe { (self.krun.krun_set_console_output)(self.ctx_id, console_c.as_ptr()) }, + "krun_set_console_output", + ) + } + + fn set_exec(&self, exec_path: &str, args: &[String], env: &[String]) -> Result<(), String> { + let exec_c = CString::new(exec_path).map_err(|e| format!("invalid exec path: {e}"))?; + let argv_slices: Vec<&str> = args.iter().map(String::as_str).collect(); + let (_argv_owners, argv_ptrs) = c_string_array(&argv_slices)?; + let env_slices: Vec<&str> = env.iter().map(String::as_str).collect(); + let (_env_owners, env_ptrs) = c_string_array(&env_slices)?; + + check( + unsafe { + (self.krun.krun_set_exec)( + self.ctx_id, + exec_c.as_ptr(), + argv_ptrs.as_ptr(), + env_ptrs.as_ptr(), + ) + }, + "krun_set_exec", + ) + } + + fn start_enter(&self) -> i32 { + unsafe { (self.krun.krun_start_enter)(self.ctx_id) } + } +} + +impl Drop for VmContext { + fn drop(&mut self) { + let ret = unsafe { (self.krun.krun_free_ctx)(self.ctx_id) }; + if ret < 0 { + eprintln!( + "warning: krun_free_ctx({}) failed with code {ret}", + self.ctx_id + ); + } + } +} + +struct GvproxyGuard { + child: Option, +} + +impl GvproxyGuard { + fn new(child: StdChild) -> Self { + Self { child: Some(child) } + } + + fn disarm(&mut self) -> Option { + self.child.take() + } +} + +impl Drop for GvproxyGuard { + fn drop(&mut self) { + if let Some(mut child) = self.child.take() { + let _ = child.kill(); + let _ = child.wait(); + } + } +} + +fn wait_for_path(path: &Path, timeout: Duration, label: &str) -> Result<(), String> { + let deadline = Instant::now() + timeout; + let mut interval = Duration::from_millis(5); + while !path.exists() { + if Instant::now() >= deadline { + return Err(format!( + "{label} did not appear within {:.1}s: {}", + timeout.as_secs_f64(), + path.display() + )); + } + std::thread::sleep(interval); + interval = (interval * 2).min(Duration::from_millis(200)); + } + Ok(()) +} + +fn hash_path_id(path: &Path) -> String { + let mut hash: u64 = 0xcbf2_9ce4_8422_2325; + for byte in path.to_string_lossy().as_bytes() { + hash ^= u64::from(*byte); + hash = hash.wrapping_mul(0x0100_0000_01b3); + } + format!("{:012x}", hash & 0x0000_ffff_ffff_ffff) +} + +fn secure_socket_base(subdir: &str) -> Result { + let base = std::env::var_os("XDG_RUNTIME_DIR").map_or_else( + || { + let fallback = PathBuf::from("/tmp"); + if fallback.is_dir() { + fallback + } else { + std::env::temp_dir() + } + }, + PathBuf::from, + ); + let dir = base.join(subdir); + + if dir.exists() { + let meta = dir + .symlink_metadata() + .map_err(|e| format!("lstat {}: {e}", dir.display()))?; + if meta.file_type().is_symlink() { + return Err(format!( + "socket directory {} is a symlink; refusing to use it", + dir.display() + )); + } + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt as _; + let uid = unsafe { libc::getuid() }; + if meta.uid() != uid { + return Err(format!( + "socket directory {} is owned by uid {} but we are uid {}", + dir.display(), + meta.uid(), + uid + )); + } + } + } else { + std::fs::create_dir_all(&dir) + .map_err(|e| format!("create socket dir {}: {e}", dir.display()))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + let _ = std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)); + } + } + + Ok(dir) +} + +fn gvproxy_socket_base(rootfs: &Path) -> Result { + Ok(secure_socket_base("osd-gv")?.join(hash_path_id(rootfs))) +} + +fn install_signal_forwarding(pid: i32) { + unsafe { + libc::signal( + libc::SIGINT, + forward_signal as *const () as libc::sighandler_t, + ); + libc::signal( + libc::SIGTERM, + forward_signal as *const () as libc::sighandler_t, + ); + } + CHILD_PID.store(pid, Ordering::Relaxed); +} + +/// Async-signal-safe handler that forwards SIGTERM to every process we +/// own: the libkrun VM worker and the gvproxy helper. We cannot rely on +/// Rust destructors (`GvproxyGuard::drop`, `ManagedDriverProcess::drop`) +/// running on signal-driven exit, so we explicitly deliver the signal +/// here. The `wait_for_child` loop reaps libkrun and `cleanup_gvproxy` +/// reaps gvproxy before `run_vm` returns. +/// +/// Only async-signal-safe libc calls are used — `kill(2)` is listed in +/// POSIX.1-2017 as async-signal-safe, atomic loads are lock-free on the +/// platforms we target. +extern "C" fn forward_signal(_sig: libc::c_int) { + let vm_pid = CHILD_PID.load(Ordering::Relaxed); + if vm_pid > 0 { + unsafe { + libc::kill(vm_pid, libc::SIGTERM); + } + } + let gv_pid = GVPROXY_PID.load(Ordering::Relaxed); + if gv_pid > 0 { + // gvproxy handles SIGTERM cleanly; no need for SIGKILL. + unsafe { + libc::kill(gv_pid, libc::SIGTERM); + } + } +} + +fn wait_for_child(pid: i32) -> Result { + let mut status: libc::c_int = 0; + let rc = unsafe { libc::waitpid(pid, &raw mut status, 0) }; + if rc < 0 { + return Err(format!( + "waitpid({pid}) failed: {}", + std::io::Error::last_os_error() + )); + } + Ok(status) +} + +fn cleanup_gvproxy(mut guard: Option) { + if let Some(mut guard) = guard.take() + && let Some(mut child) = guard.disarm() + { + let _ = child.kill(); + let _ = child.wait(); + } +} + +fn check(ret: i32, func: &'static str) -> Result<(), String> { + if ret < 0 { + Err(format!("{func} failed with error code {ret}")) + } else { + Ok(()) + } +} + +fn c_string_array(strings: &[&str]) -> Result<(Vec, Vec<*const libc::c_char>), String> { + let owned: Vec = strings + .iter() + .map(|s| CString::new(*s)) + .collect::, _>>() + .map_err(|e| format!("invalid string array entry: {e}"))?; + let mut ptrs: Vec<*const libc::c_char> = owned.iter().map(|c| c.as_ptr()).collect(); + ptrs.push(ptr::null()); + Ok((owned, ptrs)) +} + +fn path_to_cstring(path: &Path) -> Result { + let path = path + .to_str() + .ok_or_else(|| format!("path is not valid UTF-8: {}", path.display()))?; + CString::new(path).map_err(|e| format!("invalid path string {path}: {e}")) +} + +#[cfg(target_os = "linux")] +fn check_kvm_access() -> Result<(), String> { + std::fs::OpenOptions::new() + .read(true) + .open("/dev/kvm") + .map(|_| ()) + .map_err(|e| { + format!("cannot open /dev/kvm: {e}\nKVM access is required to run microVMs on Linux.") + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn qemu_config() -> VmLaunchConfig { + VmLaunchConfig { + rootfs: PathBuf::from("/rootfs"), + vcpus: 2, + mem_mib: 2048, + exec_path: "/srv/openshell-vm-sandbox-init.sh".to_string(), + args: Vec::new(), + env: vec!["OPENSHELL_ENDPOINT=http://10.0.128.1:8080".to_string()], + workdir: "/".to_string(), + log_level: 0, + console_output: PathBuf::from("/console.log"), + backend: VmBackend::Qemu, + gpu_bdf: Some("0000:01:00.0".to_string()), + tap_device: Some("vmtap-test".to_string()), + guest_ip: Some("10.0.128.2".to_string()), + host_ip: Some("10.0.128.1".to_string()), + vsock_cid: Some(4), + guest_mac: Some("02:00:00:00:00:01".to_string()), + gateway_port: Some(8080), + } + } + + #[test] + fn qemu_guest_env_vars_include_driver_runtime_metadata() { + let env = qemu_guest_env_vars(&qemu_config(), Some("1.1.1.1".to_string())); + + assert!(env.contains(&"OPENSHELL_ENDPOINT=http://10.0.128.1:8080".to_string())); + assert!(env.contains(&"VM_NET_IP=10.0.128.2".to_string())); + assert!(env.contains(&"VM_NET_GW=10.0.128.1".to_string())); + assert!(env.contains(&"VM_NET_DNS=1.1.1.1".to_string())); + assert!(env.contains(&"GPU_ENABLED=true".to_string())); + } + + #[test] + fn kernel_cmdline_keeps_guest_init_metadata_out_of_proc_cmdline() { + let cmdline = build_kernel_cmdline(&qemu_config()); + + assert!(cmdline.contains("ip=10.0.128.2::10.0.128.1:255.255.255.252:sandbox::off")); + assert!(cmdline.contains("firmware_class.path=/lib/firmware")); + assert!(!cmdline.contains("VM_NET_IP=")); + assert!(!cmdline.contains("VM_NET_GW=")); + assert!(!cmdline.contains("VM_NET_DNS=")); + assert!(!cmdline.contains("GPU_ENABLED=")); + } +} diff --git a/crates/openshell-ocsf/src/format/shorthand.rs b/crates/openshell-ocsf/src/format/shorthand.rs index 7e2296de9d..42b30fbae9 100644 --- a/crates/openshell-ocsf/src/format/shorthand.rs +++ b/crates/openshell-ocsf/src/format/shorthand.rs @@ -50,7 +50,6 @@ pub fn severity_char(severity_id: u8) -> char { #[must_use] pub fn severity_tag(severity_id: u8) -> &'static str { match severity_id { - 1 => "[INFO]", 2 => "[LOW]", 3 => "[MED]", 4 => "[HIGH]", diff --git a/crates/openshell-ocsf/src/tracing_layers/jsonl_layer.rs b/crates/openshell-ocsf/src/tracing_layers/jsonl_layer.rs index 1f7022ef8e..920483700a 100644 --- a/crates/openshell-ocsf/src/tracing_layers/jsonl_layer.rs +++ b/crates/openshell-ocsf/src/tracing_layers/jsonl_layer.rs @@ -60,10 +60,10 @@ where } // If an enabled flag is set and it reads `false`, skip writing. - if let Some(ref flag) = self.enabled { - if !flag.load(Ordering::Relaxed) { - return; - } + if let Some(ref flag) = self.enabled + && !flag.load(Ordering::Relaxed) + { + return; } if let Some(ocsf_event) = clone_current_event() diff --git a/crates/openshell-ocsf/src/tracing_layers/shorthand_layer.rs b/crates/openshell-ocsf/src/tracing_layers/shorthand_layer.rs index ea75cf0dc1..dee9848ad4 100644 --- a/crates/openshell-ocsf/src/tracing_layers/shorthand_layer.rs +++ b/crates/openshell-ocsf/src/tracing_layers/shorthand_layer.rs @@ -94,6 +94,20 @@ impl tracing::field::Visit for MessageVisitor<'_> { } } +/// Test helper: wraps `Arc>>` so it implements `Write + Send`. +#[cfg(test)] +struct SyncWriter(std::sync::Arc>>); + +#[cfg(test)] +impl Write for SyncWriter { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + self.0.lock().unwrap().write(buf) + } + fn flush(&mut self) -> std::io::Result<()> { + self.0.lock().unwrap().flush() + } +} + #[cfg(test)] mod tests { use super::*; @@ -139,17 +153,3 @@ mod tests { ); } } - -/// Test helper: wraps `Arc>>` so it implements `Write + Send`. -#[cfg(test)] -struct SyncWriter(std::sync::Arc>>); - -#[cfg(test)] -impl Write for SyncWriter { - fn write(&mut self, buf: &[u8]) -> std::io::Result { - self.0.lock().unwrap().write(buf) - } - fn flush(&mut self) -> std::io::Result<()> { - self.0.lock().unwrap().flush() - } -} diff --git a/crates/openshell-policy/src/compose.rs b/crates/openshell-policy/src/compose.rs new file mode 100644 index 0000000000..427fa4cda5 --- /dev/null +++ b/crates/openshell-policy/src/compose.rs @@ -0,0 +1,144 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Policy layer composition helpers. + +use openshell_core::proto::{NetworkPolicyRule, SandboxPolicy}; + +#[derive(Debug, Clone, PartialEq)] +pub struct ProviderPolicyLayer { + pub rule_name: String, + pub rule: NetworkPolicyRule, +} + +#[must_use] +pub fn provider_rule_name(provider_name: &str) -> String { + let sanitized = provider_name + .chars() + .map(|c| { + if c.is_ascii_alphanumeric() || c == '_' { + c.to_ascii_lowercase() + } else { + '_' + } + }) + .collect::() + .trim_matches('_') + .to_string(); + + if sanitized.is_empty() { + "_provider_unnamed".to_string() + } else { + format!("_provider_{sanitized}") + } +} + +/// Compose a normal sandbox policy from user-authored policy plus provider +/// policy layers. +/// +/// The returned policy is derived data. It preserves the source policy's +/// static fields and user-authored network policies, then concatenates each +/// provider rule under a reserved `_provider_*` key. Existing user keys are not +/// overwritten; a numeric suffix is added if needed. +#[must_use] +pub fn compose_effective_policy( + source_policy: &SandboxPolicy, + provider_layers: &[ProviderPolicyLayer], +) -> SandboxPolicy { + let mut effective = source_policy.clone(); + + for layer in provider_layers { + let key = unique_provider_rule_key(&effective, &layer.rule_name); + let mut rule = layer.rule.clone(); + if rule.name.is_empty() { + rule.name.clone_from(&key); + } + effective.network_policies.insert(key, rule); + } + + effective +} + +fn unique_provider_rule_key(policy: &SandboxPolicy, preferred: &str) -> String { + if !policy.network_policies.contains_key(preferred) { + return preferred.to_string(); + } + + for suffix in 2_u32.. { + let candidate = format!("{preferred}_{suffix}"); + if !policy.network_policies.contains_key(&candidate) { + return candidate; + } + } + + unreachable!("unbounded suffix search must find an unused provider policy key") +} + +#[cfg(test)] +mod tests { + use super::{ProviderPolicyLayer, compose_effective_policy, provider_rule_name}; + use openshell_core::proto::{NetworkEndpoint, NetworkPolicyRule, SandboxPolicy}; + + fn rule(name: &str, host: &str) -> NetworkPolicyRule { + NetworkPolicyRule { + name: name.to_string(), + endpoints: vec![NetworkEndpoint { + host: host.to_string(), + port: 443, + protocol: "rest".to_string(), + tls: String::new(), + enforcement: "enforce".to_string(), + access: "read-write".to_string(), + rules: Vec::new(), + allowed_ips: Vec::new(), + ports: Vec::new(), + deny_rules: Vec::new(), + allow_encoded_slash: false, + ..Default::default() + }], + binaries: Vec::new(), + } + } + + #[test] + fn provider_rule_name_sanitizes_provider_names() { + assert_eq!(provider_rule_name("my-github"), "_provider_my_github"); + assert_eq!(provider_rule_name("Work GitHub!"), "_provider_work_github"); + assert_eq!(provider_rule_name("..."), "_provider_unnamed"); + } + + #[test] + fn compose_concatenates_provider_rules_without_overwriting_user_rules() { + let mut source = SandboxPolicy::default(); + source.network_policies.insert( + "custom_github".to_string(), + rule("custom_github", "api.github.com"), + ); + source.network_policies.insert( + "_provider_work_github".to_string(), + rule("_provider_work_github", "example.com"), + ); + + let effective = compose_effective_policy( + &source, + &[ProviderPolicyLayer { + rule_name: "_provider_work_github".to_string(), + rule: rule("_provider_work_github", "github.com"), + }], + ); + + assert!(effective.network_policies.contains_key("custom_github")); + assert!( + effective + .network_policies + .contains_key("_provider_work_github") + ); + assert!( + effective + .network_policies + .contains_key("_provider_work_github_2") + ); + assert_eq!(source.network_policies.len(), 2); + assert_eq!(effective.network_policies.len(), 3); + } +} diff --git a/crates/openshell-policy/src/lib.rs b/crates/openshell-policy/src/lib.rs index e3c26061a1..61df0aadb8 100644 --- a/crates/openshell-policy/src/lib.rs +++ b/crates/openshell-policy/src/lib.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Shared sandbox policy parsing and defaults for OpenShell. +//! Shared sandbox policy parsing and defaults for `OpenShell`. //! //! Provides bidirectional YAML↔proto conversion for sandbox policies. //! @@ -9,17 +9,27 @@ //! policy schema. Both parsing (YAML→proto) and serialization (proto→YAML) use //! these types, ensuring round-trip fidelity. +mod compose; +mod merge; + use std::collections::{BTreeMap, HashMap}; use std::fmt; use std::path::Path; use miette::{IntoDiagnostic, Result, WrapErr}; use openshell_core::proto::{ - FilesystemPolicy, L7Allow, L7DenyRule, L7QueryMatcher, L7Rule, LandlockPolicy, NetworkBinary, - NetworkEndpoint, NetworkPolicyRule, ProcessPolicy, SandboxPolicy, + FilesystemPolicy, GraphqlOperation, L7Allow, L7DenyRule, L7QueryMatcher, L7Rule, + LandlockPolicy, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, ProcessPolicy, + SandboxPolicy, }; use serde::{Deserialize, Serialize}; +pub use compose::{ProviderPolicyLayer, compose_effective_policy, provider_rule_name}; +pub use merge::{ + PolicyMergeError, PolicyMergeOp, PolicyMergeResult, PolicyMergeWarning, generated_rule_name, + merge_policy, +}; + // --------------------------------------------------------------------------- // YAML serde types (canonical — used for both parsing and serialization) // --------------------------------------------------------------------------- @@ -81,6 +91,8 @@ struct NetworkPolicyRuleDef { struct NetworkEndpointDef { #[serde(default, skip_serializing_if = "String::is_empty")] host: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + path: String, /// Single port (backwards compat). Mutually exclusive with `ports`. /// Uses `u16` to reject invalid values >65535 at parse time. #[serde(default, skip_serializing_if = "is_zero")] @@ -102,12 +114,43 @@ struct NetworkEndpointDef { allowed_ips: Vec, #[serde(default, skip_serializing_if = "Vec::is_empty")] deny_rules: Vec, + /// When true, percent-encoded `/` (`%2F`) is preserved in path segments + /// rather than rejected by the L7 path canonicalizer. Required for + /// upstreams like GitLab that embed `%2F` in namespaced resource paths. + /// Defaults to false (strict). + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + allow_encoded_slash: bool, + #[serde(default, skip_serializing_if = "String::is_empty")] + persisted_queries: String, + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + graphql_persisted_queries: BTreeMap, + #[serde(default, skip_serializing_if = "is_zero_u32")] + graphql_max_body_bytes: u32, } +// Signature dictated by serde's `skip_serializing_if`, which requires `&T`. +#[allow(clippy::trivially_copy_pass_by_ref)] fn is_zero(v: &u16) -> bool { *v == 0 } +// Signature dictated by serde's `skip_serializing_if`, which requires `&T`. +#[allow(clippy::trivially_copy_pass_by_ref)] +fn is_zero_u32(v: &u32) -> bool { + *v == 0 +} + +#[derive(Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct GraphqlOperationDef { + #[serde(default, skip_serializing_if = "String::is_empty")] + operation_type: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + operation_name: String, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + fields: Vec, +} + #[derive(Debug, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct L7RuleDef { @@ -125,6 +168,12 @@ struct L7AllowDef { command: String, #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] query: BTreeMap, + #[serde(default, skip_serializing_if = "String::is_empty")] + operation_type: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + operation_name: String, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + fields: Vec, } #[derive(Debug, Serialize, Deserialize)] @@ -152,6 +201,12 @@ struct L7DenyRuleDef { command: String, #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] query: BTreeMap, + #[serde(default, skip_serializing_if = "String::is_empty")] + operation_type: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + operation_name: String, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + fields: Vec, } #[derive(Debug, Serialize, Deserialize)] @@ -194,6 +249,7 @@ fn to_proto(raw: PolicyFile) -> SandboxPolicy { }; NetworkEndpoint { host: e.host, + path: e.path, port: normalized_ports.first().copied().unwrap_or(0), ports: normalized_ports, protocol: e.protocol, @@ -208,6 +264,9 @@ fn to_proto(raw: PolicyFile) -> SandboxPolicy { method: r.allow.method, path: r.allow.path, command: r.allow.command, + operation_type: r.allow.operation_type, + operation_name: r.allow.operation_name, + fields: r.allow.fields, query: r .allow .query @@ -236,6 +295,9 @@ fn to_proto(raw: PolicyFile) -> SandboxPolicy { method: d.method, path: d.path, command: d.command, + operation_type: d.operation_type, + operation_name: d.operation_name, + fields: d.fields, query: d .query .into_iter() @@ -254,6 +316,23 @@ fn to_proto(raw: PolicyFile) -> SandboxPolicy { .collect(), }) .collect(), + allow_encoded_slash: e.allow_encoded_slash, + persisted_queries: e.persisted_queries, + graphql_persisted_queries: e + .graphql_persisted_queries + .into_iter() + .map(|(key, op)| { + ( + key, + GraphqlOperation { + operation_type: op.operation_type, + operation_name: op.operation_name, + fields: op.fields, + }, + ) + }) + .collect(), + graphql_max_body_bytes: e.graphql_max_body_bytes, } }) .collect(), @@ -335,6 +414,7 @@ fn from_proto(policy: &SandboxPolicy) -> PolicyFile { }; NetworkEndpointDef { host: e.host.clone(), + path: e.path.clone(), port, ports, protocol: e.protocol.clone(), @@ -351,16 +431,19 @@ fn from_proto(policy: &SandboxPolicy) -> PolicyFile { method: a.method, path: a.path, command: a.command, + operation_type: a.operation_type, + operation_name: a.operation_name, + fields: a.fields, query: a .query .into_iter() .map(|(key, matcher)| { - let yaml_matcher = if !matcher.any.is_empty() { + let yaml_matcher = if matcher.any.is_empty() { + QueryMatcherDef::Glob(matcher.glob) + } else { QueryMatcherDef::Any(QueryAnyDef { any: matcher.any, }) - } else { - QueryMatcherDef::Glob(matcher.glob) }; (key, yaml_matcher) }) @@ -377,22 +460,42 @@ fn from_proto(policy: &SandboxPolicy) -> PolicyFile { method: d.method.clone(), path: d.path.clone(), command: d.command.clone(), + operation_type: d.operation_type.clone(), + operation_name: d.operation_name.clone(), + fields: d.fields.clone(), query: d .query .iter() .map(|(key, matcher)| { - let yaml_matcher = if !matcher.any.is_empty() { + let yaml_matcher = if matcher.any.is_empty() { + QueryMatcherDef::Glob(matcher.glob.clone()) + } else { QueryMatcherDef::Any(QueryAnyDef { any: matcher.any.clone(), }) - } else { - QueryMatcherDef::Glob(matcher.glob.clone()) }; (key.clone(), yaml_matcher) }) .collect(), }) .collect(), + allow_encoded_slash: e.allow_encoded_slash, + persisted_queries: e.persisted_queries.clone(), + graphql_persisted_queries: e + .graphql_persisted_queries + .iter() + .map(|(key, op)| { + ( + key.clone(), + GraphqlOperationDef { + operation_type: op.operation_type.clone(), + operation_name: op.operation_name.clone(), + fields: op.fields.clone(), + }, + ) + }) + .collect(), + graphql_max_body_bytes: e.graphql_max_body_bytes, } }) .collect(), @@ -520,9 +623,7 @@ pub fn restrictive_default_policy() -> SandboxPolicy { /// the required `"sandbox"` value. Call this before validation so that /// policies without an explicit process section get the correct default. pub fn ensure_sandbox_process_identity(policy: &mut SandboxPolicy) { - let process = policy - .process - .get_or_insert_with(|| ProcessPolicy::default()); + let process = policy.process.get_or_insert_with(ProcessPolicy::default); if process.run_as_user.is_empty() { process.run_as_user = "sandbox".into(); } @@ -797,7 +898,7 @@ network_policies: /// Verify that the network policy `name` field survives the round-trip. #[test] fn round_trip_preserves_policy_name() { - let yaml = r#" + let yaml = r" version: 1 network_policies: my_api: @@ -807,7 +908,7 @@ network_policies: port: 443 binaries: - path: /usr/bin/curl -"#; +"; let proto1 = parse_sandbox_policy(yaml).expect("parse failed"); assert_eq!(proto1.network_policies["my_api"].name, "my-custom-api-name"); @@ -876,7 +977,7 @@ network_policies: #[test] fn parse_policy_with_network_rules() { - let yaml = r#" + let yaml = r" version: 1 network_policies: test: @@ -885,7 +986,7 @@ network_policies: - { host: example.com, port: 443 } binaries: - { path: /usr/bin/curl } -"#; +"; let policy = parse_sandbox_policy(yaml).expect("should parse"); assert_eq!(policy.network_policies.len(), 1); let rule = &policy.network_policies["test"]; @@ -1262,7 +1363,7 @@ network_policies: #[test] fn parse_ports_array() { - let yaml = r#" + let yaml = r" version: 1 network_policies: test: @@ -1271,7 +1372,7 @@ network_policies: - { host: api.example.com, ports: [80, 443] } binaries: - { path: /usr/bin/curl } -"#; +"; let policy = parse_sandbox_policy(yaml).expect("should parse"); let ep = &policy.network_policies["test"].endpoints[0]; assert_eq!(ep.ports, vec![80, 443]); @@ -1281,7 +1382,7 @@ network_policies: #[test] fn parse_single_port_normalized_to_ports() { - let yaml = r#" + let yaml = r" version: 1 network_policies: test: @@ -1290,7 +1391,7 @@ network_policies: - { host: api.example.com, port: 443 } binaries: - { path: /usr/bin/curl } -"#; +"; let policy = parse_sandbox_policy(yaml).expect("should parse"); let ep = &policy.network_policies["test"].endpoints[0]; assert_eq!(ep.ports, vec![443]); @@ -1298,9 +1399,37 @@ network_policies: } #[test] - fn round_trip_preserves_multi_port() { + fn round_trip_preserves_endpoint_path() { let yaml = r#" version: 1 +network_policies: + test: + name: test + endpoints: + - host: api.example.com + port: 443 + path: "/graphql" + protocol: graphql + rules: + - allow: + operation_type: query + binaries: + - { path: /usr/bin/curl } +"#; + let proto1 = parse_sandbox_policy(yaml).expect("parse failed"); + let yaml_out = serialize_sandbox_policy(&proto1).expect("serialize failed"); + let proto2 = parse_sandbox_policy(&yaml_out).expect("re-parse failed"); + + let ep1 = &proto1.network_policies["test"].endpoints[0]; + let ep2 = &proto2.network_policies["test"].endpoints[0]; + assert_eq!(ep1.path, "/graphql"); + assert_eq!(ep1.path, ep2.path); + } + + #[test] + fn round_trip_preserves_multi_port() { + let yaml = r" +version: 1 network_policies: test: name: test @@ -1311,7 +1440,7 @@ network_policies: - 443 binaries: - { path: /usr/bin/curl } -"#; +"; let proto1 = parse_sandbox_policy(yaml).expect("parse failed"); let yaml_out = serialize_sandbox_policy(&proto1).expect("serialize failed"); let proto2 = parse_sandbox_policy(&yaml_out).expect("re-parse failed"); @@ -1324,7 +1453,7 @@ network_policies: #[test] fn serialize_single_port_uses_compact_form() { - let yaml = r#" + let yaml = r" version: 1 network_policies: test: @@ -1333,7 +1462,7 @@ network_policies: - { host: api.example.com, port: 443 } binaries: - { path: /usr/bin/curl } -"#; +"; let proto = parse_sandbox_policy(yaml).expect("parse failed"); let yaml_out = serialize_sandbox_policy(&proto).expect("serialize failed"); // Should use compact `port: 443` form, not `ports: [443]` @@ -1476,9 +1605,60 @@ network_policies: assert_eq!(deny.query["type"].any, vec!["admin-*", "root-*"]); } + #[test] + fn round_trip_preserves_graphql_policy_fields() { + let yaml = r" +version: 1 +network_policies: + github_graphql: + name: github_graphql + endpoints: + - host: api.github.com + port: 443 + protocol: graphql + enforcement: enforce + persisted_queries: allow_registered + graphql_max_body_bytes: 131072 + graphql_persisted_queries: + abc123: + operation_type: query + operation_name: Viewer + fields: [viewer] + rules: + - allow: + operation_type: query + fields: [viewer, repository] + - allow: + operation_type: mutation + operation_name: Issue* + fields: [createIssue] + deny_rules: + - operation_type: mutation + fields: [deleteRepository] + binaries: + - path: /usr/bin/curl +"; + let proto1 = parse_sandbox_policy(yaml).expect("parse failed"); + let yaml_out = serialize_sandbox_policy(&proto1).expect("serialize failed"); + let proto2 = parse_sandbox_policy(&yaml_out).expect("re-parse failed"); + + let ep = &proto2.network_policies["github_graphql"].endpoints[0]; + assert_eq!(ep.protocol, "graphql"); + assert_eq!(ep.persisted_queries, "allow_registered"); + assert_eq!(ep.graphql_max_body_bytes, 131_072); + assert_eq!( + ep.graphql_persisted_queries["abc123"].operation_type, + "query" + ); + assert_eq!(ep.rules[0].allow.as_ref().unwrap().operation_type, "query"); + assert_eq!(ep.rules[1].allow.as_ref().unwrap().operation_name, "Issue*"); + assert_eq!(ep.deny_rules[0].operation_type, "mutation"); + assert_eq!(ep.deny_rules[0].fields, vec!["deleteRepository"]); + } + #[test] fn parse_rejects_unknown_fields_in_deny_rule() { - let yaml = r#" + let yaml = r" version: 1 network_policies: test: @@ -1489,20 +1669,20 @@ network_policies: - method: POST path: /foo bogus: true -"#; +"; assert!(parse_sandbox_policy(yaml).is_err()); } #[test] fn rejects_port_above_65535() { - let yaml = r#" + let yaml = r" version: 1 network_policies: test: endpoints: - host: example.com port: 70000 -"#; +"; assert!( parse_sandbox_policy(yaml).is_err(), "port >65535 should fail to parse" diff --git a/crates/openshell-policy/src/merge.rs b/crates/openshell-policy/src/merge.rs new file mode 100644 index 0000000000..7a5dec9166 --- /dev/null +++ b/crates/openshell-policy/src/merge.rs @@ -0,0 +1,1028 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::collections::{HashMap, HashSet}; + +use openshell_core::proto::{ + L7Allow, L7DenyRule, L7Rule, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, SandboxPolicy, +}; + +#[derive(Debug, Clone, PartialEq)] +pub enum PolicyMergeOp { + AddRule { + rule_name: String, + rule: NetworkPolicyRule, + }, + RemoveEndpoint { + rule_name: Option, + host: String, + port: u32, + }, + RemoveRule { + rule_name: String, + }, + AddDenyRules { + host: String, + port: u32, + deny_rules: Vec, + }, + AddAllowRules { + host: String, + port: u32, + rules: Vec, + }, + RemoveBinary { + rule_name: String, + binary_path: String, + }, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum PolicyMergeWarning { + ExistingProtocolRetained { + host: String, + port: u32, + existing: String, + incoming: String, + }, + ExistingEnforcementRetained { + host: String, + port: u32, + existing: String, + incoming: String, + }, + ExistingTlsRetained { + host: String, + port: u32, + existing: String, + incoming: String, + }, + ExistingAccessRetained { + host: String, + port: u32, + existing: String, + incoming: String, + }, + ExpandedAccessPreset { + host: String, + port: u32, + access: String, + }, + IgnoredIncomingAccessBecauseRulesExist { + host: String, + port: u32, + incoming: String, + }, +} + +impl std::fmt::Display for PolicyMergeWarning { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::ExistingProtocolRetained { + host, + port, + existing, + incoming, + } => write!( + f, + "endpoint {host}:{port} keeps existing protocol '{existing}' and ignores incoming '{incoming}'" + ), + Self::ExistingEnforcementRetained { + host, + port, + existing, + incoming, + } => write!( + f, + "endpoint {host}:{port} keeps existing enforcement '{existing}' and ignores incoming '{incoming}'" + ), + Self::ExistingTlsRetained { + host, + port, + existing, + incoming, + } => write!( + f, + "endpoint {host}:{port} keeps existing tls mode '{existing}' and ignores incoming '{incoming}'" + ), + Self::ExistingAccessRetained { + host, + port, + existing, + incoming, + } => write!( + f, + "endpoint {host}:{port} keeps existing access preset '{existing}' and ignores incoming '{incoming}'" + ), + Self::ExpandedAccessPreset { host, port, access } => write!( + f, + "expanded access preset '{access}' to explicit rules for endpoint {host}:{port}" + ), + Self::IgnoredIncomingAccessBecauseRulesExist { + host, + port, + incoming, + } => write!( + f, + "endpoint {host}:{port} already uses explicit rules; incoming access preset '{incoming}' was ignored" + ), + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum PolicyMergeError { + MissingRuleNameForAddRule, + InvalidEndpointReference { + host: String, + port: u32, + }, + EndpointNotFound { + host: String, + port: u32, + }, + EndpointHasNoL7Inspection { + host: String, + port: u32, + }, + UnsupportedEndpointProtocol { + host: String, + port: u32, + protocol: String, + }, + EndpointHasNoAllowBase { + host: String, + port: u32, + }, + UnsupportedAccessPreset { + host: String, + port: u32, + access: String, + }, +} + +impl std::fmt::Display for PolicyMergeError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::MissingRuleNameForAddRule => write!(f, "add-rule operation requires a rule name"), + Self::InvalidEndpointReference { host, port } => { + write!(f, "invalid endpoint reference '{host}:{port}'") + } + Self::EndpointNotFound { host, port } => { + write!( + f, + "endpoint {host}:{port} was not found in the current policy" + ) + } + Self::EndpointHasNoL7Inspection { host, port } => write!( + f, + "endpoint {host}:{port} has no L7 inspection configured (protocol is empty)" + ), + Self::UnsupportedEndpointProtocol { + host, + port, + protocol, + } => write!( + f, + "endpoint {host}:{port} uses unsupported protocol '{protocol}'; this operation currently supports only protocol 'rest'" + ), + Self::EndpointHasNoAllowBase { host, port } => write!( + f, + "endpoint {host}:{port} has no base allow set; configure access or explicit allow rules before adding deny rules" + ), + Self::UnsupportedAccessPreset { host, port, access } => write!( + f, + "endpoint {host}:{port} uses unsupported access preset '{access}'" + ), + } + } +} + +impl std::error::Error for PolicyMergeError {} + +#[derive(Debug, Clone, PartialEq)] +pub struct PolicyMergeResult { + pub policy: SandboxPolicy, + pub warnings: Vec, + pub changed: bool, +} + +pub fn merge_policy( + policy: SandboxPolicy, + operations: &[PolicyMergeOp], +) -> Result { + let mut merged = policy.clone(); + let mut warnings = Vec::new(); + + for operation in operations { + apply_operation(&mut merged, operation, &mut warnings)?; + } + + let changed = merged != policy; + Ok(PolicyMergeResult { + policy: merged, + warnings, + changed, + }) +} + +pub fn generated_rule_name(host: &str, port: u32) -> String { + let sanitized = host + .replace(['.', '-'], "_") + .chars() + .filter(|c| c.is_alphanumeric() || *c == '_') + .collect::(); + format!("allow_{sanitized}_{port}") +} + +fn apply_operation( + policy: &mut SandboxPolicy, + operation: &PolicyMergeOp, + warnings: &mut Vec, +) -> Result<(), PolicyMergeError> { + match operation { + PolicyMergeOp::AddRule { rule_name, rule } => { + add_rule(policy, rule_name, rule, warnings)?; + } + PolicyMergeOp::RemoveEndpoint { + rule_name, + host, + port, + } => { + remove_endpoint(policy, rule_name.as_deref(), host, *port); + } + PolicyMergeOp::RemoveRule { rule_name } => { + policy.network_policies.remove(rule_name); + } + PolicyMergeOp::AddDenyRules { + host, + port, + deny_rules, + } => { + let endpoint = find_endpoint_mut(policy, host, *port).ok_or_else(|| { + PolicyMergeError::EndpointNotFound { + host: host.clone(), + port: *port, + } + })?; + ensure_rest_endpoint(endpoint, host, *port)?; + if endpoint.access.is_empty() && endpoint.rules.is_empty() { + return Err(PolicyMergeError::EndpointHasNoAllowBase { + host: host.clone(), + port: *port, + }); + } + append_unique_deny_rules(&mut endpoint.deny_rules, deny_rules); + } + PolicyMergeOp::AddAllowRules { host, port, rules } => { + let endpoint = find_endpoint_mut(policy, host, *port).ok_or_else(|| { + PolicyMergeError::EndpointNotFound { + host: host.clone(), + port: *port, + } + })?; + ensure_rest_endpoint(endpoint, host, *port)?; + expand_existing_access(endpoint, host, *port, warnings)?; + append_unique_l7_rules(&mut endpoint.rules, rules); + } + PolicyMergeOp::RemoveBinary { + rule_name, + binary_path, + } => { + let should_remove = if let Some(rule) = policy.network_policies.get_mut(rule_name) { + let original_len = rule.binaries.len(); + rule.binaries.retain(|binary| binary.path != *binary_path); + original_len != rule.binaries.len() && rule.binaries.is_empty() + } else { + false + }; + if should_remove { + policy.network_policies.remove(rule_name); + } + } + } + Ok(()) +} + +fn add_rule( + policy: &mut SandboxPolicy, + rule_name: &str, + incoming_rule: &NetworkPolicyRule, + warnings: &mut Vec, +) -> Result<(), PolicyMergeError> { + if rule_name.trim().is_empty() { + return Err(PolicyMergeError::MissingRuleNameForAddRule); + } + + let mut incoming_rule = incoming_rule.clone(); + normalize_rule(&mut incoming_rule); + if incoming_rule.name.is_empty() { + incoming_rule.name = rule_name.to_string(); + } + + let target_key = if policy.network_policies.contains_key(rule_name) { + Some(rule_name.to_string()) + } else { + let mut keys: Vec<_> = policy.network_policies.keys().cloned().collect(); + keys.sort(); + keys.into_iter().find(|key| { + policy + .network_policies + .get(key) + .is_some_and(|existing_rule| rules_share_endpoint(existing_rule, &incoming_rule)) + }) + }; + + if let Some(key) = target_key { + let existing_rule = policy + .network_policies + .get_mut(&key) + .expect("existing rule must be present"); + merge_rules(existing_rule, &incoming_rule, warnings)?; + } else { + policy + .network_policies + .insert(rule_name.to_string(), incoming_rule); + } + + Ok(()) +} + +fn merge_rules( + existing_rule: &mut NetworkPolicyRule, + incoming_rule: &NetworkPolicyRule, + warnings: &mut Vec, +) -> Result<(), PolicyMergeError> { + append_unique_binaries(&mut existing_rule.binaries, &incoming_rule.binaries); + + for incoming_endpoint in &incoming_rule.endpoints { + let mut incoming_endpoint = incoming_endpoint.clone(); + normalize_endpoint(&mut incoming_endpoint); + if let Some(existing_endpoint) = + find_matching_endpoint_mut(&mut existing_rule.endpoints, &incoming_endpoint) + { + merge_endpoint(existing_endpoint, &incoming_endpoint, warnings)?; + } else { + existing_rule.endpoints.push(incoming_endpoint); + } + } + + Ok(()) +} + +fn merge_endpoint( + existing: &mut NetworkEndpoint, + incoming: &NetworkEndpoint, + warnings: &mut Vec, +) -> Result<(), PolicyMergeError> { + let host = if existing.host.is_empty() { + incoming.host.clone() + } else { + existing.host.clone() + }; + let port = canonical_ports(existing) + .into_iter() + .next() + .or_else(|| canonical_ports(incoming).into_iter().next()) + .unwrap_or(0); + + if existing.host.is_empty() { + existing.host.clone_from(&incoming.host); + } + if existing.path.is_empty() { + existing.path.clone_from(&incoming.path); + } + + merge_endpoint_ports(existing, incoming); + let existing_protocol = existing.protocol.clone(); + merge_string_field( + &mut existing.protocol, + &incoming.protocol, + PolicyMergeWarning::ExistingProtocolRetained { + host: host.clone(), + port, + existing: existing_protocol, + incoming: incoming.protocol.clone(), + }, + warnings, + ); + let existing_enforcement = existing.enforcement.clone(); + merge_string_field( + &mut existing.enforcement, + &incoming.enforcement, + PolicyMergeWarning::ExistingEnforcementRetained { + host: host.clone(), + port, + existing: existing_enforcement, + incoming: incoming.enforcement.clone(), + }, + warnings, + ); + let existing_tls = existing.tls.clone(); + merge_string_field( + &mut existing.tls, + &incoming.tls, + PolicyMergeWarning::ExistingTlsRetained { + host: host.clone(), + port, + existing: existing_tls, + incoming: incoming.tls.clone(), + }, + warnings, + ); + + if !incoming.rules.is_empty() { + expand_existing_access(existing, &host, port, warnings)?; + append_unique_l7_rules(&mut existing.rules, &incoming.rules); + if !incoming.access.is_empty() { + warnings.push(PolicyMergeWarning::IgnoredIncomingAccessBecauseRulesExist { + host, + port, + incoming: incoming.access.clone(), + }); + } + } else if !incoming.access.is_empty() { + if !existing.rules.is_empty() { + warnings.push(PolicyMergeWarning::IgnoredIncomingAccessBecauseRulesExist { + host, + port, + incoming: incoming.access.clone(), + }); + } else if existing.access.is_empty() { + existing.access.clone_from(&incoming.access); + } else if existing.access != incoming.access { + warnings.push(PolicyMergeWarning::ExistingAccessRetained { + host, + port, + existing: existing.access.clone(), + incoming: incoming.access.clone(), + }); + } + } + + append_unique_deny_rules(&mut existing.deny_rules, &incoming.deny_rules); + append_unique_strings(&mut existing.allowed_ips, &incoming.allowed_ips); + normalize_endpoint(existing); + Ok(()) +} + +fn merge_string_field( + existing: &mut String, + incoming: &str, + warning: PolicyMergeWarning, + warnings: &mut Vec, +) { + if incoming.is_empty() { + return; + } + if existing.is_empty() { + *existing = incoming.to_string(); + } else if *existing != incoming { + warnings.push(warning); + } +} + +fn merge_endpoint_ports(existing: &mut NetworkEndpoint, incoming: &NetworkEndpoint) { + let mut ports = canonical_ports(existing); + for port in canonical_ports(incoming) { + if !ports.contains(&port) { + ports.push(port); + } + } + ports.sort_unstable(); + ports.dedup(); + existing.port = ports.first().copied().unwrap_or(0); + existing.ports = ports; +} + +fn rules_share_endpoint( + existing_rule: &NetworkPolicyRule, + incoming_rule: &NetworkPolicyRule, +) -> bool { + incoming_rule.endpoints.iter().any(|incoming_endpoint| { + existing_rule + .endpoints + .iter() + .any(|existing_endpoint| endpoints_overlap(existing_endpoint, incoming_endpoint)) + }) +} + +fn endpoints_overlap(left: &NetworkEndpoint, right: &NetworkEndpoint) -> bool { + if !left.host.eq_ignore_ascii_case(&right.host) { + return false; + } + if left.path != right.path { + return false; + } + + let left_ports = canonical_ports(left); + let right_ports = canonical_ports(right); + left_ports.iter().any(|port| right_ports.contains(port)) +} + +fn canonical_ports(endpoint: &NetworkEndpoint) -> Vec { + if !endpoint.ports.is_empty() { + endpoint.ports.clone() + } else if endpoint.port > 0 { + vec![endpoint.port] + } else { + vec![] + } +} + +fn find_matching_endpoint_mut<'a>( + endpoints: &'a mut [NetworkEndpoint], + target: &NetworkEndpoint, +) -> Option<&'a mut NetworkEndpoint> { + endpoints + .iter_mut() + .find(|endpoint| endpoints_overlap(endpoint, target)) +} + +fn find_endpoint_mut<'a>( + policy: &'a mut SandboxPolicy, + host: &str, + port: u32, +) -> Option<&'a mut NetworkEndpoint> { + let mut keys: Vec<_> = policy.network_policies.keys().cloned().collect(); + keys.sort(); + let target_key = keys.into_iter().find(|key| { + policy.network_policies.get(key).is_some_and(|rule| { + rule.endpoints + .iter() + .any(|endpoint| endpoint_matches_host_port(endpoint, host, port)) + }) + })?; + + policy + .network_policies + .get_mut(&target_key) + .and_then(|rule| { + rule.endpoints + .iter_mut() + .find(|endpoint| endpoint_matches_host_port(endpoint, host, port)) + }) +} + +fn endpoint_matches_host_port(endpoint: &NetworkEndpoint, host: &str, port: u32) -> bool { + endpoint.host.eq_ignore_ascii_case(host) && canonical_ports(endpoint).contains(&port) +} + +fn ensure_rest_endpoint( + endpoint: &NetworkEndpoint, + host: &str, + port: u32, +) -> Result<(), PolicyMergeError> { + if endpoint.protocol.is_empty() { + return Err(PolicyMergeError::EndpointHasNoL7Inspection { + host: host.to_string(), + port, + }); + } + if endpoint.protocol != "rest" { + return Err(PolicyMergeError::UnsupportedEndpointProtocol { + host: host.to_string(), + port, + protocol: endpoint.protocol.clone(), + }); + } + Ok(()) +} + +fn expand_existing_access( + endpoint: &mut NetworkEndpoint, + host: &str, + port: u32, + warnings: &mut Vec, +) -> Result<(), PolicyMergeError> { + if endpoint.access.is_empty() { + return Ok(()); + } + + let access = endpoint.access.clone(); + let expanded = + expand_access_preset(&access).ok_or_else(|| PolicyMergeError::UnsupportedAccessPreset { + host: host.to_string(), + port, + access: access.clone(), + })?; + endpoint.access.clear(); + append_unique_l7_rules(&mut endpoint.rules, &expanded); + warnings.push(PolicyMergeWarning::ExpandedAccessPreset { + host: host.to_string(), + port, + access, + }); + Ok(()) +} + +fn expand_access_preset(access: &str) -> Option> { + let methods = match access { + "read-only" => vec!["GET", "HEAD", "OPTIONS"], + "read-write" => vec!["GET", "HEAD", "OPTIONS", "POST", "PUT", "PATCH"], + "full" => vec!["*"], + _ => return None, + }; + + Some( + methods + .into_iter() + .map(|method| L7Rule { + allow: Some(L7Allow { + method: method.to_string(), + path: "**".to_string(), + command: String::new(), + query: HashMap::default(), + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), + }), + }) + .collect(), + ) +} + +fn append_unique_binaries(existing: &mut Vec, incoming: &[NetworkBinary]) { + let mut seen: HashSet = existing.iter().map(|binary| binary.path.clone()).collect(); + for binary in incoming { + if seen.insert(binary.path.clone()) { + existing.push(binary.clone()); + } + } +} + +fn append_unique_strings(existing: &mut Vec, incoming: &[String]) { + let mut seen: HashSet = existing.iter().cloned().collect(); + for value in incoming { + if seen.insert(value.clone()) { + existing.push(value.clone()); + } + } +} + +fn append_unique_l7_rules(existing: &mut Vec, incoming: &[L7Rule]) { + for rule in incoming { + if !existing.contains(rule) { + existing.push(rule.clone()); + } + } +} + +fn append_unique_deny_rules(existing: &mut Vec, incoming: &[L7DenyRule]) { + for rule in incoming { + if !existing.contains(rule) { + existing.push(rule.clone()); + } + } +} + +fn normalize_rule(rule: &mut NetworkPolicyRule) { + for endpoint in &mut rule.endpoints { + normalize_endpoint(endpoint); + } + dedup_binaries(&mut rule.binaries); +} + +fn normalize_endpoint(endpoint: &mut NetworkEndpoint) { + let mut ports = canonical_ports(endpoint); + ports.sort_unstable(); + ports.dedup(); + endpoint.port = ports.first().copied().unwrap_or(0); + endpoint.ports = ports; + dedup_strings(&mut endpoint.allowed_ips); + dedup_l7_rules(&mut endpoint.rules); + dedup_deny_rules(&mut endpoint.deny_rules); +} + +fn dedup_strings(values: &mut Vec) { + let mut seen = HashSet::new(); + values.retain(|value| seen.insert(value.clone())); +} + +fn dedup_binaries(values: &mut Vec) { + let mut seen = HashSet::new(); + values.retain(|binary| seen.insert(binary.path.clone())); +} + +fn dedup_l7_rules(values: &mut Vec) { + let mut deduped = Vec::with_capacity(values.len()); + for value in std::mem::take(values) { + if !deduped.contains(&value) { + deduped.push(value); + } + } + *values = deduped; +} + +fn dedup_deny_rules(values: &mut Vec) { + let mut deduped = Vec::with_capacity(values.len()); + for value in std::mem::take(values) { + if !deduped.contains(&value) { + deduped.push(value); + } + } + *values = deduped; +} + +fn remove_endpoint(policy: &mut SandboxPolicy, rule_name: Option<&str>, host: &str, port: u32) { + let target_keys: Vec = if let Some(rule_name) = rule_name { + if policy.network_policies.contains_key(rule_name) { + vec![rule_name.to_string()] + } else { + vec![] + } + } else { + let mut keys: Vec<_> = policy.network_policies.keys().cloned().collect(); + keys.sort(); + keys + }; + + let mut empty_rules = Vec::new(); + for key in target_keys { + if let Some(rule) = policy.network_policies.get_mut(&key) { + rule.endpoints.retain_mut(|endpoint| { + if !endpoint_matches_host_port(endpoint, host, port) { + return true; + } + + let mut remaining_ports = canonical_ports(endpoint); + remaining_ports.retain(|existing_port| *existing_port != port); + remaining_ports.sort_unstable(); + remaining_ports.dedup(); + + if remaining_ports.is_empty() { + return false; + } + + endpoint.port = remaining_ports[0]; + endpoint.ports = remaining_ports; + true + }); + + if rule.endpoints.is_empty() { + empty_rules.push(key); + } + } + } + + for key in empty_rules { + policy.network_policies.remove(&key); + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use super::{ + PolicyMergeError, PolicyMergeOp, PolicyMergeWarning, generated_rule_name, merge_policy, + }; + use crate::restrictive_default_policy; + use openshell_core::proto::{ + L7Allow, L7DenyRule, L7Rule, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, + }; + + fn endpoint(host: &str, port: u32) -> NetworkEndpoint { + NetworkEndpoint { + host: host.to_string(), + port, + ports: vec![port], + ..Default::default() + } + } + + fn rule_with_endpoint(name: &str, host: &str, port: u32) -> NetworkPolicyRule { + NetworkPolicyRule { + name: name.to_string(), + endpoints: vec![endpoint(host, port)], + ..Default::default() + } + } + + fn rest_rule(method: &str, path: &str) -> L7Rule { + L7Rule { + allow: Some(L7Allow { + method: method.to_string(), + path: path.to_string(), + command: String::new(), + query: HashMap::new(), + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), + }), + } + } + + #[test] + fn generated_rule_name_sanitizes_host() { + assert_eq!( + generated_rule_name("api.github.com", 443), + "allow_api_github_com_443" + ); + } + + #[test] + fn add_rule_merges_l7_fields_into_existing_endpoint() { + let mut policy = restrictive_default_policy(); + policy.network_policies.insert( + "existing".to_string(), + NetworkPolicyRule { + name: "existing".to_string(), + endpoints: vec![endpoint("api.github.com", 443)], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }, + ); + + let incoming = NetworkPolicyRule { + name: "incoming".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.github.com".to_string(), + port: 443, + ports: vec![443], + protocol: "rest".to_string(), + enforcement: "enforce".to_string(), + rules: vec![rest_rule("GET", "/repos/**")], + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/gh".to_string(), + ..Default::default() + }], + }; + + let result = merge_policy( + policy, + &[PolicyMergeOp::AddRule { + rule_name: "allow_api_github_com_443".to_string(), + rule: incoming, + }], + ) + .expect("merge should succeed"); + + let rule = &result.policy.network_policies["existing"]; + let endpoint = &rule.endpoints[0]; + assert_eq!(endpoint.protocol, "rest"); + assert_eq!(endpoint.enforcement, "enforce"); + assert_eq!(endpoint.rules.len(), 1); + assert_eq!(rule.binaries.len(), 2); + } + + #[test] + fn add_allow_expands_access_preset() { + let mut policy = restrictive_default_policy(); + policy.network_policies.insert( + "github".to_string(), + NetworkPolicyRule { + name: "github".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.github.com".to_string(), + port: 443, + ports: vec![443], + protocol: "rest".to_string(), + access: "read-only".to_string(), + ..Default::default() + }], + ..Default::default() + }, + ); + + let result = merge_policy( + policy, + &[PolicyMergeOp::AddAllowRules { + host: "api.github.com".to_string(), + port: 443, + rules: vec![rest_rule("POST", "/repos/*/issues")], + }], + ) + .expect("merge should succeed"); + + let endpoint = &result.policy.network_policies["github"].endpoints[0]; + assert!(endpoint.access.is_empty()); + assert_eq!(endpoint.rules.len(), 4); + assert!(result.warnings.iter().any(|warning| matches!( + warning, + PolicyMergeWarning::ExpandedAccessPreset { access, .. } if access == "read-only" + ))); + } + + #[test] + fn add_deny_requires_rest_protocol() { + let mut policy = restrictive_default_policy(); + policy.network_policies.insert( + "db".to_string(), + NetworkPolicyRule { + name: "db".to_string(), + endpoints: vec![NetworkEndpoint { + host: "db.example.com".to_string(), + port: 5432, + ports: vec![5432], + protocol: "sql".to_string(), + access: "full".to_string(), + ..Default::default() + }], + ..Default::default() + }, + ); + + let error = merge_policy( + policy, + &[PolicyMergeOp::AddDenyRules { + host: "db.example.com".to_string(), + port: 5432, + deny_rules: vec![L7DenyRule { + method: "POST".to_string(), + path: "/admin".to_string(), + ..Default::default() + }], + }], + ) + .expect_err("merge should fail"); + + assert!(matches!( + error, + PolicyMergeError::UnsupportedEndpointProtocol { protocol, .. } if protocol == "sql" + )); + } + + #[test] + fn remove_endpoint_drops_only_requested_port() { + let mut policy = restrictive_default_policy(); + policy.network_policies.insert( + "multi".to_string(), + NetworkPolicyRule { + name: "multi".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.example.com".to_string(), + port: 80, + ports: vec![80, 443], + ..Default::default() + }], + ..Default::default() + }, + ); + + let result = merge_policy( + policy, + &[PolicyMergeOp::RemoveEndpoint { + rule_name: None, + host: "api.example.com".to_string(), + port: 443, + }], + ) + .expect("merge should succeed"); + + let endpoint = &result.policy.network_policies["multi"].endpoints[0]; + assert_eq!(endpoint.ports, vec![80]); + assert_eq!(endpoint.port, 80); + } + + #[test] + fn remove_binary_removes_rule_when_last_binary_is_deleted() { + let mut policy = restrictive_default_policy(); + policy.network_policies.insert( + "github".to_string(), + NetworkPolicyRule { + name: "github".to_string(), + endpoints: vec![endpoint("api.github.com", 443)], + binaries: vec![NetworkBinary { + path: "/usr/bin/gh".to_string(), + ..Default::default() + }], + }, + ); + + let result = merge_policy( + policy, + &[PolicyMergeOp::RemoveBinary { + rule_name: "github".to_string(), + binary_path: "/usr/bin/gh".to_string(), + }], + ) + .expect("merge should succeed"); + + assert!(!result.policy.network_policies.contains_key("github")); + } + + #[test] + fn add_rule_without_existing_match_inserts_requested_key() { + let policy = restrictive_default_policy(); + let result = merge_policy( + policy, + &[PolicyMergeOp::AddRule { + rule_name: "allow_api_example_com_443".to_string(), + rule: rule_with_endpoint("custom", "api.example.com", 443), + }], + ) + .expect("merge should succeed"); + + assert!( + result + .policy + .network_policies + .contains_key("allow_api_example_com_443") + ); + } +} diff --git a/crates/openshell-prover/src/lib.rs b/crates/openshell-prover/src/lib.rs index feff7d1d54..82922253de 100644 --- a/crates/openshell-prover/src/lib.rs +++ b/crates/openshell-prover/src/lib.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Formal policy verification for OpenShell sandboxes. +//! Formal policy verification for `OpenShell` sandboxes. //! //! Encodes sandbox policies, binary capabilities, and credential scopes as Z3 //! SMT constraints, then checks reachability queries to detect data exfiltration @@ -91,7 +91,7 @@ mod tests { #[test] fn test_parse_policy() { let path = testdata_dir().join("policy.yaml"); - let model = policy::parse_policy(&path).expect("failed to parse policy"); + let model = parse_policy(&path).expect("failed to parse policy"); assert_eq!(model.version, 1); assert!(model.network_policies.contains_key("github_api")); let rule = &model.network_policies["github_api"]; @@ -104,7 +104,7 @@ mod tests { #[test] fn test_filesystem_policy() { let path = testdata_dir().join("policy.yaml"); - let model = policy::parse_policy(&path).expect("failed to parse policy"); + let model = parse_policy(&path).expect("failed to parse policy"); let readable = model.filesystem_policy.readable_paths(); assert!(readable.contains(&"/usr".to_owned())); assert!(readable.contains(&"/sandbox".to_owned())); @@ -114,12 +114,12 @@ mod tests { // 3. Workdir NOT included by default (matches runtime behavior). #[test] fn test_include_workdir_default() { - let yaml = r#" + let yaml = r" version: 1 filesystem_policy: read_only: - /usr -"#; +"; let model = policy::parse_policy_str(yaml).expect("parse"); let readable = model.filesystem_policy.readable_paths(); assert!(!readable.contains(&"/sandbox".to_owned())); @@ -128,13 +128,13 @@ filesystem_policy: // 4. Workdir excluded when include_workdir: false. #[test] fn test_include_workdir_false() { - let yaml = r#" + let yaml = r" version: 1 filesystem_policy: include_workdir: false read_only: - /usr -"#; +"; let model = policy::parse_policy_str(yaml).expect("parse"); let readable = model.filesystem_policy.readable_paths(); assert!(!readable.contains(&"/sandbox".to_owned())); @@ -143,14 +143,14 @@ filesystem_policy: // 5. No duplicate when workdir already in read_write. #[test] fn test_include_workdir_no_duplicate() { - let yaml = r#" + let yaml = r" version: 1 filesystem_policy: include_workdir: true read_write: - /sandbox - /tmp -"#; +"; let model = policy::parse_policy_str(yaml).expect("parse"); let readable = model.filesystem_policy.readable_paths(); let sandbox_count = readable.iter().filter(|p| *p == "/sandbox").count(); @@ -163,12 +163,12 @@ filesystem_policy: let policy_path = testdata_dir().join("policy.yaml"); let creds_path = testdata_dir().join("credentials.yaml"); - let pol = policy::parse_policy(&policy_path).expect("parse policy"); + let pol = parse_policy(&policy_path).expect("parse policy"); let cred_set = credentials::load_credential_set_embedded(&creds_path).expect("load creds"); let bin_reg = registry::load_embedded_binary_registry().expect("load registry"); - let z3_model = model::build_model(pol, cred_set, bin_reg); - let findings = queries::run_all_queries(&z3_model); + let z3_model = build_model(pol, cred_set, bin_reg); + let findings = run_all_queries(&z3_model); let query_types: std::collections::HashSet<&str> = findings.iter().map(|f| f.query.as_str()).collect(); @@ -195,12 +195,12 @@ filesystem_policy: let policy_path = testdata_dir().join("empty-policy.yaml"); let creds_path = testdata_dir().join("credentials.yaml"); - let pol = policy::parse_policy(&policy_path).expect("parse policy"); + let pol = parse_policy(&policy_path).expect("parse policy"); let cred_set = credentials::load_credential_set_embedded(&creds_path).expect("load creds"); let bin_reg = registry::load_embedded_binary_registry().expect("load registry"); - let z3_model = model::build_model(pol, cred_set, bin_reg); - let findings = queries::run_all_queries(&z3_model); + let z3_model = build_model(pol, cred_set, bin_reg); + let findings = run_all_queries(&z3_model); assert!( findings.is_empty(), diff --git a/crates/openshell-prover/src/model.rs b/crates/openshell-prover/src/model.rs index b96e7fef2c..bf52993d47 100644 --- a/crates/openshell-prover/src/model.rs +++ b/crates/openshell-prover/src/model.rs @@ -27,7 +27,7 @@ impl EndpointId { } } -/// Z3-backed reachability model for an OpenShell sandbox policy. +/// Z3-backed reachability model for an `OpenShell` sandbox policy. pub struct ReachabilityModel { pub policy: PolicyModel, pub credentials: CredentialSet, @@ -169,18 +169,18 @@ impl ReachabilityModel { allowed.iter().any(|m| write_set.contains(m.as_str())) }; - let l7w_var = Bool::new_const(format!("l7_allows_write_{ek}")); + let l7_write_var = Bool::new_const(format!("l7_allows_write_{ek}")); if ep.is_l7_enforced() { if has_write { - self.solver.assert(&l7w_var); + self.solver.assert(&l7_write_var); } else { - self.solver.assert(&!l7w_var.clone()); + self.solver.assert(&!l7_write_var.clone()); } } else { // L4-only: all methods pass through - self.solver.assert(&l7w_var); + self.solver.assert(&l7_write_var); } - self.l7_allows_write.insert(ek, l7w_var); + self.l7_allows_write.insert(ek, l7_write_var); } } } @@ -256,13 +256,14 @@ impl ReachabilityModel { } self.credential_has_write.insert(host.clone(), cw_var); - let cd_var = Bool::new_const(format!("credential_has_destructive_{host}")); + let destructive_var = Bool::new_const(format!("credential_has_destructive_{host}")); if has_destructive { - self.solver.assert(&cd_var); + self.solver.assert(&destructive_var); } else { - self.solver.assert(&!cd_var.clone()); + self.solver.assert(&!destructive_var.clone()); } - self.credential_has_destructive.insert(host.clone(), cd_var); + self.credential_has_destructive + .insert(host.clone(), destructive_var); } } @@ -364,7 +365,7 @@ impl ReachabilityModel { has_access, exfil, Bool::or(&[ - Bool::and(&[!l7_enforced.clone(), http.clone()]), + Bool::and(&[!l7_enforced, http.clone()]), Bool::and(&[l7_write, http]), bypass, ]), diff --git a/crates/openshell-prover/src/policy.rs b/crates/openshell-prover/src/policy.rs index b116fd5b09..8aea4b7d00 100644 --- a/crates/openshell-prover/src/policy.rs +++ b/crates/openshell-prover/src/policy.rs @@ -249,23 +249,13 @@ pub struct NetworkPolicyRule { } /// Filesystem access policy. -#[derive(Debug, Clone)] +#[derive(Debug, Clone, Default)] pub struct FilesystemPolicy { pub include_workdir: bool, pub read_only: Vec, pub read_write: Vec, } -impl Default for FilesystemPolicy { - fn default() -> Self { - Self { - include_workdir: false, - read_only: Vec::new(), - read_write: Vec::new(), - } - } -} - impl FilesystemPolicy { /// All readable paths (union of `read_only` and `read_write`), with workdir /// added when `include_workdir` is true and not already present. @@ -302,7 +292,7 @@ impl Default for PolicyModel { } impl PolicyModel { - /// All (policy_name, endpoint) pairs. + /// All (`policy_name`, endpoint) pairs. pub fn all_endpoints(&self) -> Vec<(&str, &Endpoint)> { let mut result = Vec::new(); for (name, rule) in &self.network_policies { @@ -327,7 +317,7 @@ impl PolicyModel { result } - /// All (binary, policy_name, endpoint) triples. + /// All (binary, `policy_name`, endpoint) triples. pub fn binary_endpoint_pairs(&self) -> Vec<(&Binary, &str, &Endpoint)> { let mut result = Vec::new(); for (name, rule) in &self.network_policies { @@ -345,7 +335,7 @@ impl PolicyModel { // Parsing // --------------------------------------------------------------------------- -/// Parse an OpenShell policy YAML file into a [`PolicyModel`]. +/// Parse an `OpenShell` policy YAML file into a [`PolicyModel`]. pub fn parse_policy(path: &Path) -> Result { let contents = std::fs::read_to_string(path) .into_diagnostic() diff --git a/crates/openshell-prover/src/queries.rs b/crates/openshell-prover/src/queries.rs index 001255ceae..6a0c7f6a66 100644 --- a/crates/openshell-prover/src/queries.rs +++ b/crates/openshell-prover/src/queries.rs @@ -44,8 +44,7 @@ pub fn check_data_exfiltration(model: &ReachabilityModel) -> Vec { ( "l4_only".to_owned(), format!( - "L4-only endpoint — no HTTP inspection, {} can send arbitrary data", - bpath + "L4-only endpoint — no HTTP inspection, {bpath} can send arbitrary data" ), ) } else { diff --git a/crates/openshell-prover/src/registry.rs b/crates/openshell-prover/src/registry.rs index 3acd52f828..63a3fce3bc 100644 --- a/crates/openshell-prover/src/registry.rs +++ b/crates/openshell-prover/src/registry.rs @@ -129,7 +129,7 @@ impl BinaryCapability { /// Whether the binary can perform write actions. pub fn can_write(&self) -> bool { - self.protocols.iter().any(|p| p.can_write()) || self.can_construct_http + self.protocols.iter().any(BinaryProtocol::can_write) || self.can_construct_http } /// Short mechanisms by which this binary can write. @@ -170,12 +170,11 @@ impl BinaryRegistry { return cap.clone(); } for (reg_path, cap) in &self.binaries { - if reg_path.contains('*') { - if let Ok(pattern) = glob::Pattern::new(reg_path) { - if pattern.matches(path) { - return cap.clone(); - } - } + if reg_path.contains('*') + && let Ok(pattern) = glob::Pattern::new(reg_path) + && pattern.matches(path) + { + return cap.clone(); } } BinaryCapability { diff --git a/crates/openshell-providers/Cargo.toml b/crates/openshell-providers/Cargo.toml index 41f9ed6c0f..1a3bda8f67 100644 --- a/crates/openshell-providers/Cargo.toml +++ b/crates/openshell-providers/Cargo.toml @@ -12,6 +12,8 @@ repository.workspace = true [dependencies] openshell-core = { path = "../openshell-core" } +serde = { workspace = true } +serde_yml = { workspace = true } thiserror = { workspace = true } [lints] diff --git a/crates/openshell-providers/src/lib.rs b/crates/openshell-providers/src/lib.rs index e2bcc0c097..b2bf1e2348 100644 --- a/crates/openshell-providers/src/lib.rs +++ b/crates/openshell-providers/src/lib.rs @@ -5,6 +5,7 @@ mod context; mod discovery; +mod profiles; mod providers; #[cfg(test)] mod test_helpers; @@ -16,6 +17,7 @@ pub use openshell_core::proto::Provider; pub use context::{DiscoveryContext, RealDiscoveryContext}; pub use discovery::discover_with_spec; +pub use profiles::{ProviderTypeProfile, default_profiles, get_default_profile}; #[derive(Debug, thiserror::Error)] pub enum ProviderError { @@ -115,6 +117,16 @@ impl ProviderRegistry { .map_or(&[], ProviderPlugin::credential_env_vars) } + #[must_use] + pub fn profile(&self, id: &str) -> Option<&'static ProviderTypeProfile> { + get_default_profile(id) + } + + #[must_use] + pub fn profiles(&self) -> Vec<&'static ProviderTypeProfile> { + default_profiles().iter().collect() + } + #[must_use] pub fn known_types(&self) -> Vec<&'static str> { let mut types = self.plugins.keys().copied().collect::>(); diff --git a/crates/openshell-providers/src/profiles.rs b/crates/openshell-providers/src/profiles.rs new file mode 100644 index 0000000000..b9c161d26f --- /dev/null +++ b/crates/openshell-providers/src/profiles.rs @@ -0,0 +1,361 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Declarative provider type profiles. + +#![allow(deprecated)] // NetworkBinary::harness remains in the public proto for compatibility. + +use openshell_core::proto::{ + NetworkBinary, NetworkEndpoint, NetworkPolicyRule, ProviderProfile, ProviderProfileCategory, + ProviderProfileCredential, +}; +use serde::{Deserialize, Deserializer, de}; +use std::collections::HashSet; +use std::sync::OnceLock; + +const BUILT_IN_PROFILE_YAMLS: &[&str] = &[ + include_str!("../../../providers/anthropic.yaml"), + include_str!("../../../providers/claude.yaml"), + include_str!("../../../providers/codex.yaml"), + include_str!("../../../providers/copilot.yaml"), + include_str!("../../../providers/github.yaml"), + include_str!("../../../providers/gitlab.yaml"), + include_str!("../../../providers/nvidia.yaml"), + include_str!("../../../providers/openai.yaml"), + include_str!("../../../providers/opencode.yaml"), + include_str!("../../../providers/outlook.yaml"), +]; + +#[derive(Debug, thiserror::Error)] +pub enum ProfileError { + #[error("failed to parse provider profile YAML: {0}")] + Parse(#[from] serde_yml::Error), + #[error("provider profile id is required")] + MissingId, + #[error("duplicate provider profile id: {0}")] + DuplicateId(String), + #[error("provider profile '{id}' has invalid endpoint '{host}:{port}'")] + InvalidEndpoint { id: String, host: String, port: u32 }, + #[error("provider profile '{id}' has duplicate credential env var '{env_var}'")] + DuplicateCredentialEnvVar { id: String, env_var: String }, +} + +#[derive(Debug, Clone, Deserialize, PartialEq, Eq)] +pub struct CredentialProfile { + pub name: String, + #[serde(default)] + pub description: String, + #[serde(default)] + pub env_vars: Vec, + #[serde(default)] + pub required: bool, + #[serde(default)] + pub auth_style: String, + #[serde(default)] + pub header_name: String, + #[serde(default)] + pub query_param: String, +} + +#[derive(Debug, Clone, Deserialize, PartialEq, Eq)] +pub struct EndpointProfile { + pub host: String, + pub port: u32, + #[serde(default)] + pub protocol: String, + #[serde(default)] + pub access: String, + #[serde(default)] + pub enforcement: String, +} + +#[derive(Debug, Clone, Deserialize, PartialEq, Eq)] +pub struct ProviderTypeProfile { + pub id: String, + pub display_name: String, + #[serde(default)] + pub description: String, + #[serde( + default = "default_category", + deserialize_with = "deserialize_category" + )] + pub category: ProviderProfileCategory, + #[serde(default)] + pub credentials: Vec, + #[serde(default)] + pub endpoints: Vec, + #[serde(default)] + pub binaries: Vec, + #[serde(default)] + pub inference_capable: bool, +} + +impl ProviderTypeProfile { + #[must_use] + pub fn credential_env_vars(&self) -> Vec<&str> { + let mut vars = Vec::new(); + for credential in &self.credentials { + for env_var in &credential.env_vars { + if !vars.contains(&env_var.as_str()) { + vars.push(env_var.as_str()); + } + } + } + vars + } + + #[must_use] + pub fn to_proto(&self) -> ProviderProfile { + ProviderProfile { + id: self.id.clone(), + display_name: self.display_name.clone(), + description: self.description.clone(), + category: self.category as i32, + credentials: self + .credentials + .iter() + .map(|credential| ProviderProfileCredential { + name: credential.name.clone(), + description: credential.description.clone(), + env_vars: credential.env_vars.clone(), + required: credential.required, + auth_style: credential.auth_style.clone(), + header_name: credential.header_name.clone(), + query_param: credential.query_param.clone(), + }) + .collect(), + endpoints: self.endpoints.iter().map(endpoint_to_proto).collect(), + binaries: self + .binaries + .iter() + .map(|path| NetworkBinary { + path: path.clone(), + harness: false, + }) + .collect(), + inference_capable: self.inference_capable, + } + } + + #[must_use] + pub fn network_policy_rule(&self, rule_name: &str) -> NetworkPolicyRule { + NetworkPolicyRule { + name: rule_name.to_string(), + endpoints: self.endpoints.iter().map(endpoint_to_proto).collect(), + binaries: self + .binaries + .iter() + .map(|path| NetworkBinary { + path: path.clone(), + harness: false, + }) + .collect(), + } + } +} + +fn default_category() -> ProviderProfileCategory { + ProviderProfileCategory::Other +} + +fn deserialize_category<'de, D>(deserializer: D) -> Result +where + D: Deserializer<'de>, +{ + let raw = String::deserialize(deserializer)?; + provider_profile_category_from_yaml(&raw) + .ok_or_else(|| de::Error::custom(format!("unsupported provider profile category: {raw}"))) +} + +fn provider_profile_category_from_yaml(raw: &str) -> Option { + match raw.trim().to_ascii_lowercase().replace('-', "_").as_str() { + "" | "other" => Some(ProviderProfileCategory::Other), + "inference" => Some(ProviderProfileCategory::Inference), + "agent" => Some(ProviderProfileCategory::Agent), + "source_control" => Some(ProviderProfileCategory::SourceControl), + "messaging" => Some(ProviderProfileCategory::Messaging), + "data" => Some(ProviderProfileCategory::Data), + "knowledge" => Some(ProviderProfileCategory::Knowledge), + _ => None, + } +} + +fn endpoint_to_proto(endpoint: &EndpointProfile) -> NetworkEndpoint { + NetworkEndpoint { + host: endpoint.host.clone(), + port: endpoint.port, + protocol: endpoint.protocol.clone(), + tls: String::new(), + enforcement: endpoint.enforcement.clone(), + access: endpoint.access.clone(), + rules: Vec::new(), + allowed_ips: Vec::new(), + ports: Vec::new(), + deny_rules: Vec::new(), + allow_encoded_slash: false, + ..Default::default() + } +} + +pub fn parse_profile_yaml(input: &str) -> Result { + Ok(serde_yml::from_str::(input)?) +} + +pub fn parse_profile_catalog_yamls( + inputs: &[&str], +) -> Result, ProfileError> { + let mut profiles = inputs + .iter() + .map(|input| parse_profile_yaml(input)) + .collect::, _>>()?; + validate_profiles(&profiles)?; + profiles.sort_by(|left, right| left.id.cmp(&right.id)); + Ok(profiles) +} + +fn validate_profiles(profiles: &[ProviderTypeProfile]) -> Result<(), ProfileError> { + let mut ids = HashSet::new(); + for profile in profiles { + if profile.id.trim().is_empty() { + return Err(ProfileError::MissingId); + } + if !ids.insert(profile.id.clone()) { + return Err(ProfileError::DuplicateId(profile.id.clone())); + } + + let mut env_vars = HashSet::new(); + for credential in &profile.credentials { + for env_var in &credential.env_vars { + if !env_vars.insert(env_var) { + return Err(ProfileError::DuplicateCredentialEnvVar { + id: profile.id.clone(), + env_var: env_var.clone(), + }); + } + } + } + + for endpoint in &profile.endpoints { + if endpoint.host.trim().is_empty() || endpoint.port == 0 || endpoint.port > 65_535 { + return Err(ProfileError::InvalidEndpoint { + id: profile.id.clone(), + host: endpoint.host.clone(), + port: endpoint.port, + }); + } + } + } + Ok(()) +} + +static DEFAULT_PROFILES: OnceLock> = OnceLock::new(); + +#[must_use] +pub fn default_profiles() -> &'static [ProviderTypeProfile] { + DEFAULT_PROFILES + .get_or_init(|| { + parse_profile_catalog_yamls(BUILT_IN_PROFILE_YAMLS) + .expect("built-in provider profiles must be valid YAML") + }) + .as_slice() +} + +#[must_use] +pub fn get_default_profile(id: &str) -> Option<&'static ProviderTypeProfile> { + default_profiles() + .iter() + .find(|profile| profile.id.eq_ignore_ascii_case(id)) +} + +#[cfg(test)] +mod tests { + use openshell_core::proto::ProviderProfileCategory; + + use super::{ + ProfileError, default_profiles, get_default_profile, parse_profile_catalog_yamls, + parse_profile_yaml, + }; + + #[test] + fn default_profiles_are_sorted_by_id() { + let ids = default_profiles() + .iter() + .map(|profile| profile.id.as_str()) + .collect::>(); + let mut sorted = ids.clone(); + sorted.sort_unstable(); + assert_eq!(ids, sorted); + } + + #[test] + fn github_profile_materializes_policy_metadata() { + let profile = get_default_profile("github").expect("github profile"); + let proto = profile.to_proto(); + + assert_eq!(proto.id, "github"); + assert_eq!( + proto.category, + ProviderProfileCategory::SourceControl as i32 + ); + assert_eq!(proto.endpoints.len(), 2); + assert_eq!(proto.binaries.len(), 4); + } + + #[test] + fn credential_env_vars_are_deduplicated_in_profile_order() { + let profile = get_default_profile("copilot").expect("copilot profile"); + assert_eq!( + profile.credential_env_vars(), + vec!["COPILOT_GITHUB_TOKEN", "GH_TOKEN", "GITHUB_TOKEN"] + ); + } + + #[test] + fn parse_profile_yaml_reads_single_provider_document() { + let profile = parse_profile_yaml( + r" +id: example +display_name: Example +credentials: + - name: api_key + env_vars: [EXAMPLE_API_KEY] +", + ) + .expect("profile should parse"); + + assert_eq!(profile.id, "example"); + assert_eq!(profile.category, ProviderProfileCategory::Other); + assert_eq!(profile.credential_env_vars(), vec!["EXAMPLE_API_KEY"]); + } + + #[test] + fn parse_profile_catalog_yamls_rejects_duplicate_ids() { + let err = parse_profile_catalog_yamls(&[ + r" +id: duplicate +display_name: First +", + r" +id: duplicate +display_name: Second +", + ]) + .unwrap_err(); + + assert!(matches!(err, ProfileError::DuplicateId(id) if id == "duplicate")); + } + + #[test] + fn parse_profile_catalog_yamls_rejects_invalid_endpoint_ports() { + let err = parse_profile_catalog_yamls(&[r" +id: bad-endpoint +display_name: Bad Endpoint +endpoints: + - host: api.example.com + port: 0 +"]) + .unwrap_err(); + + assert!(matches!(err, ProfileError::InvalidEndpoint { id, .. } if id == "bad-endpoint")); + } +} diff --git a/crates/openshell-router/src/backend.rs b/crates/openshell-router/src/backend.rs index 8dbae65020..88a6e213ab 100644 --- a/crates/openshell-router/src/backend.rs +++ b/crates/openshell-router/src/backend.rs @@ -4,6 +4,7 @@ use crate::RouterError; use crate::config::{AuthHeader, ResolvedRoute}; use crate::mock; +use std::collections::HashSet; #[derive(Debug, Clone, PartialEq, Eq)] pub struct ValidatedEndpoint { @@ -32,7 +33,7 @@ struct ValidationProbe { protocol: &'static str, body: bytes::Bytes, /// Alternate body to try when the primary probe fails with HTTP 400. - /// Used for OpenAI chat completions where newer models require + /// Used for `OpenAI` chat completions where newer models require /// `max_completion_tokens` while legacy/self-hosted backends only /// accept `max_tokens`. fallback_body: Option, @@ -62,6 +63,9 @@ enum StreamingBody { Buffered(Option), } +const COMMON_INFERENCE_REQUEST_HEADERS: [&str; 4] = + ["content-type", "accept", "accept-encoding", "user-agent"]; + impl StreamingProxyResponse { /// Create from a fully-buffered [`ProxyResponse`] (for mock routes). pub fn from_buffered(resp: ProxyResponse) -> Self { @@ -83,7 +87,64 @@ impl StreamingProxyResponse { } } -/// Build an HTTP request to the backend configured in `route`. +fn sanitize_request_headers( + route: &ResolvedRoute, + headers: &[(String, String)], +) -> Vec<(String, String)> { + let mut allowed = HashSet::new(); + allowed.extend( + COMMON_INFERENCE_REQUEST_HEADERS + .iter() + .map(|name| (*name).to_string()), + ); + allowed.extend( + route + .passthrough_headers + .iter() + .map(|name| name.to_ascii_lowercase()), + ); + allowed.extend( + route + .default_headers + .iter() + .map(|(name, _)| name.to_ascii_lowercase()), + ); + + headers + .iter() + .filter_map(|(name, value)| { + let name_lc = name.to_ascii_lowercase(); + if should_strip_request_header(&name_lc) || !allowed.contains(&name_lc) { + return None; + } + Some((name.clone(), value.clone())) + }) + .collect() +} + +fn should_strip_request_header(name: &str) -> bool { + matches!( + name, + "authorization" | "x-api-key" | "host" | "content-length" + ) || is_hop_by_hop_header(name) +} + +fn is_hop_by_hop_header(name: &str) -> bool { + matches!( + name, + "connection" + | "keep-alive" + | "proxy-authenticate" + | "proxy-authorization" + | "proxy-connection" + | "te" + | "trailer" + | "transfer-encoding" + | "upgrade" + ) +} + +/// Build and send an HTTP request to the backend configured in `route`. /// /// Returns the prepared [`reqwest::RequestBuilder`] with auth, headers, model /// rewrite, and body applied. The caller decides whether to apply a total @@ -97,6 +158,7 @@ fn prepare_backend_request( body: bytes::Bytes, ) -> Result<(reqwest::RequestBuilder, String), RouterError> { let url = build_backend_url(&route.endpoint, path); + let headers = sanitize_request_headers(route, headers); let reqwest_method: reqwest::Method = method .parse() @@ -113,17 +175,7 @@ fn prepare_backend_request( builder = builder.header(*header_name, &route.api_key); } } - - // Strip auth and host headers — auth is re-injected above from the route - // config, and host must match the upstream. - let strip_headers: [&str; 3] = ["authorization", "x-api-key", "host"]; - - // Forward non-sensitive headers. - for (name, value) in headers { - let name_lc = name.to_ascii_lowercase(); - if strip_headers.contains(&name_lc.as_str()) { - continue; - } + for (name, value) in &headers { builder = builder.header(name.as_str(), value.as_str()); } @@ -301,18 +353,18 @@ pub async fn verify_backend_endpoint( // there is a fallback body, retry with the alternate token parameter. // This handles the split between `max_completion_tokens` (GPT-5+) and // `max_tokens` (legacy/self-hosted backends). - if let (Err(err), Some(fallback_body)) = (&result, probe.fallback_body) { - if err.kind == ValidationFailureKind::RequestShape { - return try_validation_request( - client, - route, - probe.path, - probe.protocol, - headers, - fallback_body, - ) - .await; - } + if let (Err(err), Some(fallback_body)) = (&result, probe.fallback_body) + && err.kind == ValidationFailureKind::RequestShape + { + return try_validation_request( + client, + route, + probe.path, + probe.protocol, + headers, + fallback_body, + ) + .await; } result @@ -510,10 +562,95 @@ mod tests { protocols: protocols.iter().map(|p| (*p).to_string()).collect(), auth, default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], + passthrough_headers: vec![ + "anthropic-version".to_string(), + "anthropic-beta".to_string(), + ], timeout: crate::config::DEFAULT_ROUTE_TIMEOUT, } } + #[test] + fn sanitize_request_headers_drops_unknown_sensitive_headers() { + let route = ResolvedRoute { + name: "inference.local".to_string(), + endpoint: "https://api.example.com/v1".to_string(), + model: "test-model".to_string(), + api_key: "sk-test".to_string(), + protocols: vec!["openai_chat_completions".to_string()], + auth: AuthHeader::Bearer, + default_headers: Vec::new(), + passthrough_headers: vec!["openai-organization".to_string()], + timeout: crate::config::DEFAULT_ROUTE_TIMEOUT, + }; + + let kept = super::sanitize_request_headers( + &route, + &[ + ("content-type".to_string(), "application/json".to_string()), + ("authorization".to_string(), "Bearer client".to_string()), + ("cookie".to_string(), "session=1".to_string()), + ("x-amz-security-token".to_string(), "token".to_string()), + ("openai-organization".to_string(), "org_123".to_string()), + ], + ); + + assert!( + kept.iter() + .any(|(name, _)| name.eq_ignore_ascii_case("content-type")) + ); + assert!( + kept.iter() + .any(|(name, _)| name.eq_ignore_ascii_case("openai-organization")) + ); + assert!( + kept.iter() + .all(|(name, _)| !name.eq_ignore_ascii_case("authorization")) + ); + assert!( + kept.iter() + .all(|(name, _)| !name.eq_ignore_ascii_case("cookie")) + ); + assert!( + kept.iter() + .all(|(name, _)| !name.eq_ignore_ascii_case("x-amz-security-token")) + ); + } + + #[test] + fn sanitize_request_headers_preserves_allowed_provider_headers() { + let route = test_route( + "https://api.anthropic.com/v1", + &["anthropic_messages"], + AuthHeader::Custom("x-api-key"), + ); + + let kept = super::sanitize_request_headers( + &route, + &[ + ("anthropic-version".to_string(), "2024-10-22".to_string()), + ( + "anthropic-beta".to_string(), + "tool-use-2024-10-22".to_string(), + ), + ("x-api-key".to_string(), "client-key".to_string()), + ], + ); + + assert!(kept.iter().any( + |(name, value)| name.eq_ignore_ascii_case("anthropic-version") && value == "2024-10-22" + )); + assert!( + kept.iter() + .any(|(name, value)| name.eq_ignore_ascii_case("anthropic-beta") + && value == "tool-use-2024-10-22") + ); + assert!( + kept.iter() + .all(|(name, _)| !name.eq_ignore_ascii_case("x-api-key")) + ); + } + #[tokio::test] async fn verify_backend_endpoint_uses_route_auth_and_shape() { let mock_server = MockServer::start().await; @@ -631,7 +768,7 @@ mod tests { assert_eq!(validated.protocol, "openai_chat_completions"); } - /// Non-chat-completions probes (e.g. anthropic_messages) should not + /// Non-chat-completions probes (e.g. `anthropic_messages`) should not /// have a fallback — a 400 remains a hard failure. #[tokio::test] async fn verify_non_chat_completions_no_fallback() { diff --git a/crates/openshell-router/src/config.rs b/crates/openshell-router/src/config.rs index b531e091d9..ef5d90946b 100644 --- a/crates/openshell-router/src/config.rs +++ b/crates/openshell-router/src/config.rs @@ -34,7 +34,7 @@ pub struct RouteConfig { /// A fully-resolved route ready for the router to forward requests. /// /// The router is provider-agnostic — all provider-specific decisions -/// (auth header style, default headers, base URL) are made by the +/// (auth header style, default headers, passthrough headers, base URL) are made by the /// caller during resolution. #[derive(Clone)] pub struct ResolvedRoute { @@ -48,6 +48,8 @@ pub struct ResolvedRoute { pub auth: AuthHeader, /// Extra headers injected on every request (e.g. `anthropic-version`). pub default_headers: Vec<(String, String)>, + /// Client-supplied headers that may be forwarded to the upstream backend. + pub passthrough_headers: Vec, /// Per-request timeout for proxied inference calls. pub timeout: Duration, } @@ -62,6 +64,7 @@ impl std::fmt::Debug for ResolvedRoute { .field("protocols", &self.protocols) .field("auth", &self.auth) .field("default_headers", &self.default_headers) + .field("passthrough_headers", &self.passthrough_headers) .field("timeout", &self.timeout) .finish() } @@ -125,7 +128,8 @@ impl RouteConfig { ))); } - let (auth, default_headers) = auth_from_provider_type(self.provider_type.as_deref()); + let (auth, default_headers, passthrough_headers) = + route_headers_from_provider_type(self.provider_type.as_deref()); Ok(ResolvedRoute { name: self.name.clone(), @@ -135,17 +139,21 @@ impl RouteConfig { protocols, auth, default_headers, + passthrough_headers, timeout: DEFAULT_ROUTE_TIMEOUT, }) } } -/// Derive auth header style and default headers from a provider type string. +/// Derive auth header style, default headers, and passthrough headers from a +/// provider type string. /// -/// Delegates to [`openshell_core::inference::auth_for_provider_type`] which -/// uses the centralized `InferenceProviderProfile` registry. -fn auth_from_provider_type(provider_type: Option<&str>) -> (AuthHeader, Vec<(String, String)>) { - openshell_core::inference::auth_for_provider_type(provider_type.unwrap_or("")) +/// Delegates to [`openshell_core::inference::route_headers_for_provider_type`] +/// which uses the centralized `InferenceProviderProfile` registry. +fn route_headers_from_provider_type( + provider_type: Option<&str>, +) -> (AuthHeader, Vec<(String, String)>, Vec) { + openshell_core::inference::route_headers_for_provider_type(provider_type.unwrap_or("")) } #[cfg(test)] @@ -155,7 +163,7 @@ mod tests { #[test] fn load_from_file_valid_yaml_round_trip() { - let yaml = r#" + let yaml = r" routes: - name: inference.local endpoint: http://localhost:8000/v1 @@ -167,7 +175,7 @@ routes: model: gpt-4o protocols: [openai_chat_completions, anthropic_messages] api_key: sk-prod-key -"#; +"; let mut f = tempfile::NamedTempFile::new().unwrap(); f.write_all(yaml.as_bytes()).unwrap(); @@ -200,13 +208,13 @@ routes: #[test] fn load_from_file_missing_api_key_returns_error() { - let yaml = r#" + let yaml = r" routes: - name: inference.local endpoint: http://localhost:8000/v1 model: llama-3 protocols: [openai_chat_completions] -"#; +"; let mut f = tempfile::NamedTempFile::new().unwrap(); f.write_all(yaml.as_bytes()).unwrap(); @@ -223,15 +231,16 @@ routes: } #[test] + #[allow(unsafe_code)] // std::env::set_var/remove_var require unsafe in Rust 2024 fn load_from_file_api_key_env_resolves_from_environment() { - let yaml = r#" + let yaml = r" routes: - name: inference.local endpoint: http://localhost:8000/v1 model: llama-3 protocols: [openai_chat_completions] api_key_env: NAV_TEST_API_KEY_FOR_YAML_TEST -"#; +"; // SAFETY: this test runs single-threaded; no other thread reads this var. unsafe { std::env::set_var("NAV_TEST_API_KEY_FOR_YAML_TEST", "from-env") }; let mut f = tempfile::NamedTempFile::new().unwrap(); @@ -263,6 +272,7 @@ routes: protocols: vec!["openai_chat_completions".to_string()], auth: AuthHeader::Bearer, default_headers: Vec::new(), + passthrough_headers: Vec::new(), timeout: DEFAULT_ROUTE_TIMEOUT, }; let debug_output = format!("{route:?}"); @@ -278,22 +288,34 @@ routes: #[test] fn auth_from_anthropic_provider_uses_custom_header() { - let (auth, headers) = auth_from_provider_type(Some("anthropic")); + let (auth, headers, passthrough_headers) = + route_headers_from_provider_type(Some("anthropic")); assert_eq!(auth, AuthHeader::Custom("x-api-key")); assert!(headers.iter().any(|(k, _)| k == "anthropic-version")); + assert!( + passthrough_headers + .iter() + .any(|name| name == "anthropic-beta") + ); } #[test] fn auth_from_openai_provider_uses_bearer() { - let (auth, headers) = auth_from_provider_type(Some("openai")); + let (auth, headers, passthrough_headers) = route_headers_from_provider_type(Some("openai")); assert_eq!(auth, AuthHeader::Bearer); assert!(headers.is_empty()); + assert!( + passthrough_headers + .iter() + .any(|name| name == "openai-organization") + ); } #[test] fn auth_from_none_defaults_to_bearer() { - let (auth, headers) = auth_from_provider_type(None); + let (auth, headers, passthrough_headers) = route_headers_from_provider_type(None); assert_eq!(auth, AuthHeader::Bearer); assert!(headers.is_empty()); + assert!(passthrough_headers.is_empty()); } } diff --git a/crates/openshell-router/src/mock.rs b/crates/openshell-router/src/mock.rs index a17ce486f0..66fc804144 100644 --- a/crates/openshell-router/src/mock.rs +++ b/crates/openshell-router/src/mock.rs @@ -131,6 +131,7 @@ mod tests { protocols: protocols.iter().map(ToString::to_string).collect(), auth: crate::config::AuthHeader::Bearer, default_headers: Vec::new(), + passthrough_headers: Vec::new(), timeout: crate::config::DEFAULT_ROUTE_TIMEOUT, } } diff --git a/crates/openshell-router/tests/backend_integration.rs b/crates/openshell-router/tests/backend_integration.rs index d9aecb0e37..6b21de94dd 100644 --- a/crates/openshell-router/tests/backend_integration.rs +++ b/crates/openshell-router/tests/backend_integration.rs @@ -15,6 +15,7 @@ fn mock_candidates(base_url: &str) -> Vec { protocols: vec!["openai_chat_completions".to_string()], auth: AuthHeader::Bearer, default_headers: Vec::new(), + passthrough_headers: vec!["openai-organization".to_string(), "x-model-id".to_string()], timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }] } @@ -118,6 +119,7 @@ async fn proxy_no_compatible_route_returns_error() { protocols: vec!["anthropic_messages".to_string()], auth: AuthHeader::Custom("x-api-key"), default_headers: Vec::new(), + passthrough_headers: Vec::new(), timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }]; @@ -169,6 +171,39 @@ async fn proxy_strips_auth_header() { assert_eq!(response.status, 200); } +#[tokio::test] +async fn proxy_forwards_openai_organization_header() { + let mock_server = MockServer::start().await; + + Mock::given(method("POST")) + .and(path("/v1/chat/completions")) + .and(bearer_token("test-api-key")) + .and(header("openai-organization", "org_123")) + .respond_with(ResponseTemplate::new(200).set_body_string("{}")) + .mount(&mock_server) + .await; + + let router = Router::new().unwrap(); + let candidates = mock_candidates(&mock_server.uri()); + + let response = router + .proxy_with_candidates( + "openai_chat_completions", + "POST", + "/v1/chat/completions", + vec![ + ("openai-organization".to_string(), "org_123".to_string()), + ("cookie".to_string(), "session=abc".to_string()), + ], + bytes::Bytes::new(), + &candidates, + ) + .await + .unwrap(); + + assert_eq!(response.status, 200); +} + #[tokio::test] async fn proxy_mock_route_returns_canned_response() { let router = Router::new().unwrap(); @@ -180,6 +215,7 @@ async fn proxy_mock_route_returns_canned_response() { protocols: vec!["openai_chat_completions".to_string()], auth: AuthHeader::Bearer, default_headers: Vec::new(), + passthrough_headers: Vec::new(), timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }]; @@ -315,6 +351,10 @@ async fn proxy_uses_x_api_key_for_anthropic_route() { protocols: vec!["anthropic_messages".to_string()], auth: AuthHeader::Custom("x-api-key"), default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], + passthrough_headers: vec![ + "anthropic-version".to_string(), + "anthropic-beta".to_string(), + ], timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }]; @@ -374,6 +414,10 @@ async fn proxy_anthropic_does_not_send_bearer_auth() { protocols: vec!["anthropic_messages".to_string()], auth: AuthHeader::Custom("x-api-key"), default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], + passthrough_headers: vec![ + "anthropic-version".to_string(), + "anthropic-beta".to_string(), + ], timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }]; @@ -419,6 +463,10 @@ async fn proxy_forwards_client_anthropic_version_header() { protocols: vec!["anthropic_messages".to_string()], auth: AuthHeader::Custom("x-api-key"), default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], + passthrough_headers: vec![ + "anthropic-version".to_string(), + "anthropic-beta".to_string(), + ], timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }]; @@ -509,6 +557,7 @@ async fn streaming_proxy_completes_despite_exceeding_route_timeout() { protocols: vec!["openai_chat_completions".to_string()], auth: AuthHeader::Bearer, default_headers: Vec::new(), + passthrough_headers: Vec::new(), // Route timeout shorter than the backend delay — streaming must // NOT be constrained by this. timeout: Duration::from_secs(1), @@ -572,6 +621,7 @@ async fn buffered_proxy_enforces_route_timeout() { protocols: vec!["openai_chat_completions".to_string()], auth: AuthHeader::Bearer, default_headers: Vec::new(), + passthrough_headers: Vec::new(), timeout: Duration::from_secs(1), }]; diff --git a/crates/openshell-sandbox/Cargo.toml b/crates/openshell-sandbox/Cargo.toml index 541784ee60..4e07521cec 100644 --- a/crates/openshell-sandbox/Cargo.toml +++ b/crates/openshell-sandbox/Cargo.toml @@ -35,7 +35,7 @@ miette = { workspace = true } thiserror = { workspace = true } anyhow = { workspace = true } hmac = "0.12" -sha2 = "0.10" +sha2 = { workspace = true } hex = "0.4" russh = "0.57" rand_core = "0.6" @@ -53,6 +53,9 @@ webpki-roots = { workspace = true } # HTTP bytes = { workspace = true } +# UUID +uuid = { workspace = true } + # Encoding base64 = { workspace = true } @@ -60,8 +63,11 @@ base64 = { workspace = true } ipnet = "2" # Serialization +serde = { workspace = true } serde_json = { workspace = true } serde_yml = { workspace = true } +apollo-parser = { workspace = true } +glob = { workspace = true } # Logging tracing = { workspace = true } diff --git a/crates/openshell-sandbox/data/sandbox-policy.rego b/crates/openshell-sandbox/data/sandbox-policy.rego index 939c2e5cb1..9fa8206271 100644 --- a/crates/openshell-sandbox/data/sandbox-policy.rego +++ b/crates/openshell-sandbox/data/sandbox-policy.rego @@ -179,7 +179,7 @@ default allow_request = false _policy_allows_l7(policy) if { some ep ep := policy.endpoints[_] - endpoint_matches_request(ep, input.network) + endpoint_matches_l7_request(ep, input.network, input.request) request_allowed_for_endpoint(input.request, ep) } @@ -207,7 +207,7 @@ default deny_request = false _policy_denies_l7(policy) if { some ep ep := policy.endpoints[_] - endpoint_matches_request(ep, input.network) + endpoint_matches_l7_request(ep, input.network, input.request) request_denied_for_endpoint(input.request, ep) } @@ -239,6 +239,27 @@ request_denied_for_endpoint(request, endpoint) if { command_matches(request.command, deny_rule.command) } +# --- L7 deny rule matching: GraphQL operation --- + +request_denied_for_endpoint(request, endpoint) if { + graphql_request_has_operations(request) + some deny_rule + deny_rule := endpoint.deny_rules[_] + deny_rule.operation_type + op := request.graphql.operations[_] + graphql_deny_rule_matches_operation(op, deny_rule, endpoint) +} + +# A GraphQL endpoint path is authoritative once it matches. If the parsed +# GraphQL request is malformed, hash-only without a trusted registry entry, or +# contains an operation outside the GraphQL allow rules, a broader REST rule on +# the same host:port must not allow it through. +request_denied_for_endpoint(request, endpoint) if { + endpoint.protocol == "graphql" + is_object(request.graphql) + not graphql_request_allowed(request, endpoint) +} + # Deny query matching: fail-closed semantics. # If no query rules on the deny rule, match unconditionally (any query params). # If query rules present, trigger the deny if ANY value for a configured key @@ -286,9 +307,40 @@ deny_any_value_matches(values, matcher) if { # --- L7 deny reason --- +request_deny_reason := reason if { + input.request + graphql_request_error(input.request) + reason := sprintf("GraphQL request rejected: %s", [input.request.graphql.error]) +} + +request_deny_reason := reason if { + input.request + not graphql_request_error(input.request) + graphql_request_has_unregistered_persisted_query(input.request, matched_endpoint_config) + reason := "GraphQL persisted query is not registered" +} + +request_deny_reason := reason if { + input.request + deny_request + graphql_request_has_operations(input.request) + not graphql_request_has_unregistered_persisted_query(input.request, matched_endpoint_config) + reason := "GraphQL operation blocked by endpoint policy" +} + +request_deny_reason := reason if { + input.request + not deny_request + not allow_request + graphql_request_has_operations(input.request) + not graphql_request_has_unregistered_persisted_query(input.request, matched_endpoint_config) + reason := "GraphQL operation not permitted by policy" +} + request_deny_reason := reason if { input.request deny_request + not graphql_request_has_operations(input.request) reason := sprintf("%s %s blocked by deny rule", [input.request.method, input.request.path]) } @@ -296,6 +348,7 @@ request_deny_reason := reason if { input.request not deny_request not allow_request + not graphql_request_has_operations(input.request) reason := sprintf("%s %s not permitted by policy", [input.request.method, input.request.path]) } @@ -319,6 +372,149 @@ request_allowed_for_endpoint(request, endpoint) if { command_matches(request.command, rule.allow.command) } +# --- L7 rule matching: GraphQL operation --- + +request_allowed_for_endpoint(request, endpoint) if { + graphql_request_allowed(request, endpoint) +} + +graphql_request_allowed(request, endpoint) if { + graphql_request_has_operations(request) + not graphql_request_error(request) + not graphql_request_has_unregistered_persisted_query(request, endpoint) + not graphql_request_has_unallowed_operation(request, endpoint) +} + +graphql_request_has_operations(request) if { + is_object(request.graphql) + operations := object.get(request.graphql, "operations", []) + count(operations) > 0 +} + +graphql_request_error(request) if { + is_object(request.graphql) + error := object.get(request.graphql, "error", "") + error != "" +} + +graphql_request_has_unallowed_operation(request, endpoint) if { + op := request.graphql.operations[_] + not graphql_operation_allowed(op, endpoint) +} + +graphql_operation_allowed(op, endpoint) if { + rule := endpoint.rules[_] + rule.allow.operation_type + graphql_allow_rule_matches_operation(op, rule.allow, endpoint) +} + +graphql_request_has_unregistered_persisted_query(request, endpoint) if { + op := request.graphql.operations[_] + graphql_operation_needs_registry(op) + not graphql_registered_operation(op, endpoint) +} + +graphql_operation_needs_registry(op) if { + object.get(op, "persisted_query", false) == true + object.get(op, "operation_type", "") == "" +} + +graphql_registered_operation(op, endpoint) if { + object.get(endpoint, "persisted_queries", "deny") == "allow_registered" + id := graphql_operation_registry_key(op) + endpoint.graphql_persisted_queries[id] +} + +graphql_operation_registry_key(op) := key if { + key := object.get(op, "persisted_query_hash", "") + key != "" +} + +graphql_operation_registry_key(op) := key if { + object.get(op, "persisted_query_hash", "") == "" + key := object.get(op, "persisted_query_id", "") + key != "" +} + +graphql_effective_operation(op, endpoint) := registered if { + graphql_operation_needs_registry(op) + key := graphql_operation_registry_key(op) + registered := endpoint.graphql_persisted_queries[key] +} + +graphql_effective_operation(op, _) := op if { + not graphql_operation_needs_registry(op) +} + +graphql_allow_rule_matches_operation(op, rule, endpoint) if { + effective := graphql_effective_operation(op, endpoint) + graphql_operation_type_matches(effective, rule) + graphql_operation_name_matches(effective, rule) + graphql_allow_fields_match(effective, rule) +} + +graphql_deny_rule_matches_operation(op, rule, endpoint) if { + effective := graphql_effective_operation(op, endpoint) + graphql_operation_type_matches(effective, rule) + graphql_operation_name_matches(effective, rule) + graphql_deny_fields_match(effective, rule) +} + +graphql_operation_type_matches(_, rule) if { + object.get(rule, "operation_type", "") == "*" +} + +graphql_operation_type_matches(op, rule) if { + expected := object.get(rule, "operation_type", "") + expected != "" + expected != "*" + lower(object.get(op, "operation_type", "")) == lower(expected) +} + +graphql_operation_name_matches(_, rule) if { + object.get(rule, "operation_name", "") == "" +} + +graphql_operation_name_matches(op, rule) if { + pattern := object.get(rule, "operation_name", "") + pattern != "" + name := object.get(op, "operation_name", "") + glob.match(pattern, [], name) +} + +# Allow-side field constraints are intentionally all-selected-fields semantics: +# if a rule declares fields, every root field selected by the operation must +# match one of the rule patterns. This prevents mixed-operation requests from +# allowing an unlisted field because one safe field also appeared. +graphql_allow_fields_match(_, rule) if { + count(object.get(rule, "fields", [])) == 0 +} + +graphql_allow_fields_match(op, rule) if { + count(object.get(rule, "fields", [])) > 0 + count(object.get(op, "fields", [])) > 0 + not graphql_operation_has_unmatched_field(op, rule) +} + +graphql_operation_has_unmatched_field(op, rule) if { + field := object.get(op, "fields", [])[_] + not graphql_field_matches_any(field, object.get(rule, "fields", [])) +} + +graphql_deny_fields_match(_, rule) if { + count(object.get(rule, "fields", [])) == 0 +} + +graphql_deny_fields_match(op, rule) if { + field := object.get(op, "fields", [])[_] + graphql_field_matches_any(field, object.get(rule, "fields", [])) +} + +graphql_field_matches_any(field, patterns) if { + pattern := patterns[_] + glob.match(pattern, [], field) +} + # Wildcard "*" matches any method; otherwise case-insensitive exact match. method_matches(_, "*") if true @@ -328,6 +524,13 @@ method_matches(actual, expected) if { } # Path matching: "**" matches everything; otherwise glob.match with "/" delimiter. +# +# INVARIANT: `input.request.path` is canonicalized by the sandbox before +# policy evaluation — percent-decoded, dot-segments resolved, doubled +# slashes collapsed, `;params` stripped, `%2F` rejected (unless an +# endpoint opts in). Patterns here must therefore match canonical paths; +# do not attempt defensive matching against `..` or `%2e%2e` — those +# inputs are rejected at the L7 parser boundary before this rule runs. path_matches(_, "**") if true path_matches(actual, pattern) if { @@ -394,8 +597,8 @@ command_matches(actual, expected) if { # --- Matched endpoint config (for L7 and allowed_ips extraction) --- # Returns the raw endpoint object for the matched policy + host:port. -# Used by Rust to extract L7 config (protocol, tls, enforcement) and/or -# allowed_ips for SSRF allowlist validation. +# Used by Rust to extract L7 config (protocol, tls, enforcement, +# allow_encoded_slash) and/or allowed_ips for SSRF allowlist validation. # Per-policy helper: returns matching endpoint configs for a single policy. _policy_endpoint_configs(policy) := [ep | @@ -440,6 +643,21 @@ endpoint_matches_request(ep, network) if { ep.ports[_] == network.port } +endpoint_matches_l7_request(ep, network, request) if { + endpoint_matches_request(ep, network) + endpoint_path_matches_request(ep, request) +} + +endpoint_path_matches_request(ep, request) if { + object.get(ep, "path", "") == "" +} + +endpoint_path_matches_request(ep, request) if { + path := object.get(ep, "path", "") + path != "" + path_matches(request.path, path) +} + # An endpoint has extended config if it specifies L7 protocol, allowed_ips, # or an explicit tls mode (e.g. tls: skip). endpoint_has_extended_config(ep) if { diff --git a/crates/openshell-sandbox/src/bypass_monitor.rs b/crates/openshell-sandbox/src/bypass_monitor.rs index d0e49c42d7..1a7ec5f99f 100644 --- a/crates/openshell-sandbox/src/bypass_monitor.rs +++ b/crates/openshell-sandbox/src/bypass_monitor.rs @@ -170,17 +170,14 @@ pub fn spawn( } }; - let stdout = match child.stdout.take() { - Some(s) => s, - None => { - let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Other) - .severity(SeverityId::Low) - .message("dmesg --follow produced no stdout; bypass monitor will not run") - .build(); - ocsf_emit!(event); - return; - } + let Some(stdout) = child.stdout.take() else { + let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Other) + .severity(SeverityId::Low) + .message("dmesg --follow produced no stdout; bypass monitor will not run") + .build(); + ocsf_emit!(event); + return; }; let reader = std::io::BufReader::new(stdout); @@ -300,9 +297,59 @@ fn resolve_process_identity(entrypoint_pid: u32, src_port: u16) -> (String, Stri { use crate::procfs; - match procfs::resolve_tcp_peer_identity(entrypoint_pid, src_port) { - Ok((binary_path, pid)) => { - let ancestors = procfs::collect_ancestor_binaries(pid, entrypoint_pid); + match procfs::resolve_tcp_peer_socket_owners(entrypoint_pid, src_port) { + Ok(socket_owners) => { + let mut identities = Vec::new(); + for owner in &socket_owners.owners { + let Ok(binary_path) = procfs::binary_path(owner.pid.cast_signed()) else { + continue; + }; + let ancestors = procfs::collect_ancestor_binaries(owner.pid, entrypoint_pid); + identities.push((owner.pid, binary_path, ancestors)); + } + + if identities.is_empty() { + return ("-".to_string(), "-".to_string(), "-".to_string()); + } + + identities.sort_by_key(|(pid, _, _)| *pid); + let first_identity = (identities[0].1.clone(), identities[0].2.clone()); + let ambiguous = identities + .iter() + .skip(1) + .any(|(_, binary_path, ancestors)| { + binary_path != &first_identity.0 || ancestors != &first_identity.1 + }); + + if ambiguous { + let pids = identities + .iter() + .map(|(pid, _, _)| pid.to_string()) + .collect::>() + .join(", "); + let owner_summary = identities + .iter() + .map(|(pid, binary_path, ancestors)| { + let ancestors_str = if ancestors.is_empty() { + "-".to_string() + } else { + ancestors + .iter() + .map(|p| p.display().to_string()) + .collect::>() + .join(" -> ") + }; + format!( + "pid={pid} binary={} ancestors=[{ancestors_str}]", + binary_path.display() + ) + }) + .collect::>() + .join("; "); + return ("ambiguous".to_string(), pids, owner_summary); + } + + let (pid, binary_path, ancestors) = identities.remove(0); let ancestors_str = if ancestors.is_empty() { "-".to_string() } else { @@ -445,6 +492,88 @@ mod tests { assert!(hint_for_event(&event).contains("UDP")); } + #[cfg(target_os = "linux")] + #[test] + fn resolve_process_identity_surfaces_ambiguous_shared_socket() { + use std::ffi::CString; + use std::net::{TcpListener, TcpStream}; + use std::os::fd::AsRawFd; + use std::time::{Duration, Instant}; + + if !std::path::Path::new("/bin/sleep").exists() { + eprintln!("skipping: /bin/sleep not available"); + return; + } + + let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); + let listener_port = listener.local_addr().unwrap().port(); + let stream = TcpStream::connect(("127.0.0.1", listener_port)).expect("connect"); + let peer_port = stream.local_addr().unwrap().port(); + let (_accepted, _) = listener.accept().expect("accept"); + + let fd = stream.as_raw_fd(); + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + unsafe { + let flags = libc::fcntl(fd, libc::F_GETFD); + assert!(flags >= 0, "F_GETFD failed"); + assert_eq!( + libc::fcntl(fd, libc::F_SETFD, flags & !libc::FD_CLOEXEC), + 0, + "F_SETFD failed" + ); + } + + let sleep_path = CString::new("/bin/sleep").unwrap(); + let arg0 = CString::new("sleep").unwrap(); + let arg1 = CString::new("30").unwrap(); + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + let child_pid = unsafe { libc::fork() }; + assert!(child_pid >= 0, "fork failed"); + if child_pid == 0 { + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + unsafe { + libc::execl( + sleep_path.as_ptr(), + arg0.as_ptr(), + arg1.as_ptr(), + std::ptr::null::(), + ); + libc::_exit(127); + } + } + + let deadline = Instant::now() + Duration::from_secs(2); + loop { + if let Ok(link) = std::fs::read_link(format!("/proc/{child_pid}/exe")) + && link.to_string_lossy().contains("sleep") + { + break; + } + assert!( + Instant::now() < deadline, + "child pid {child_pid} did not exec into sleep within 2s" + ); + std::thread::sleep(Duration::from_millis(20)); + } + + let (binary, pid, ancestors) = resolve_process_identity(std::process::id(), peer_port); + + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + unsafe { + libc::kill(child_pid, libc::SIGKILL); + libc::waitpid(child_pid, std::ptr::null_mut(), 0); + } + + assert_eq!(binary, "ambiguous"); + assert!(pid.contains(&std::process::id().to_string())); + assert!(pid.contains(&child_pid.to_string())); + assert!(ancestors.contains("binary=")); + } + #[test] fn extract_field_basic() { let s = "DST=1.2.3.4 LEN=60"; diff --git a/crates/openshell-sandbox/src/child_env.rs b/crates/openshell-sandbox/src/child_env.rs index ebd47e2251..e764afdfef 100644 --- a/crates/openshell-sandbox/src/child_env.rs +++ b/crates/openshell-sandbox/src/child_env.rs @@ -5,7 +5,7 @@ use std::path::Path; const LOCAL_NO_PROXY: &str = "127.0.0.1,localhost,::1"; -pub(crate) fn proxy_env_vars(proxy_url: &str) -> [(&'static str, String); 9] { +pub fn proxy_env_vars(proxy_url: &str) -> [(&'static str, String); 9] { [ ("ALL_PROXY", proxy_url.to_owned()), ("HTTP_PROXY", proxy_url.to_owned()), @@ -21,17 +21,20 @@ pub(crate) fn proxy_env_vars(proxy_url: &str) -> [(&'static str, String); 9] { ] } -pub(crate) fn tls_env_vars( +pub fn tls_env_vars( ca_cert_path: &Path, combined_bundle_path: &Path, -) -> [(&'static str, String); 4] { +) -> [(&'static str, String); 5] { let ca_cert_path = ca_cert_path.display().to_string(); let combined_bundle_path = combined_bundle_path.display().to_string(); [ - ("NODE_EXTRA_CA_CERTS", ca_cert_path.clone()), + ("NODE_EXTRA_CA_CERTS", ca_cert_path), ("SSL_CERT_FILE", combined_bundle_path.clone()), ("REQUESTS_CA_BUNDLE", combined_bundle_path.clone()), - ("CURL_CA_BUNDLE", combined_bundle_path), + ("CURL_CA_BUNDLE", combined_bundle_path.clone()), + // Ubuntu Noble's git links against libcurl-gnutls, which ignores SSL_CERT_FILE. + // git reads GIT_SSL_CAINFO (or http.sslCAInfo) to locate the CA bundle. + ("GIT_SSL_CAINFO", combined_bundle_path), ] } @@ -79,5 +82,8 @@ mod tests { assert!(stdout.contains("NODE_EXTRA_CA_CERTS=/etc/openshell-tls/openshell-ca.pem")); assert!(stdout.contains("SSL_CERT_FILE=/etc/openshell-tls/ca-bundle.pem")); + assert!(stdout.contains("REQUESTS_CA_BUNDLE=/etc/openshell-tls/ca-bundle.pem")); + assert!(stdout.contains("CURL_CA_BUNDLE=/etc/openshell-tls/ca-bundle.pem")); + assert!(stdout.contains("GIT_SSL_CAINFO=/etc/openshell-tls/ca-bundle.pem")); } } diff --git a/crates/openshell-sandbox/src/denial_aggregator.rs b/crates/openshell-sandbox/src/denial_aggregator.rs index 175f31afc8..d64be7f1d1 100644 --- a/crates/openshell-sandbox/src/denial_aggregator.rs +++ b/crates/openshell-sandbox/src/denial_aggregator.rs @@ -221,6 +221,5 @@ pub struct FlushableL7Sample { fn current_time_ms() -> i64 { SystemTime::now() .duration_since(UNIX_EPOCH) - .map(|d| i64::try_from(d.as_millis()).unwrap_or(i64::MAX)) - .unwrap_or(0) + .map_or(0, |d| i64::try_from(d.as_millis()).unwrap_or(i64::MAX)) } diff --git a/crates/openshell-sandbox/src/grpc_client.rs b/crates/openshell-sandbox/src/grpc_client.rs index 5503637eec..44f3723554 100644 --- a/crates/openshell-sandbox/src/grpc_client.rs +++ b/crates/openshell-sandbox/src/grpc_client.rs @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 //! gRPC client for fetching sandbox policy, provider environment, and inference -//! route bundles from OpenShell server. +//! route bundles from `OpenShell` server. use std::collections::HashMap; use std::time::Duration; @@ -14,10 +14,11 @@ use openshell_core::proto::{ SandboxPolicy as ProtoSandboxPolicy, SubmitPolicyAnalysisRequest, UpdateConfigRequest, inference_client::InferenceClient, open_shell_client::OpenShellClient, }; +use tonic::service::interceptor::InterceptedService; use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity}; use tracing::debug; -/// Create a channel to the OpenShell server. +/// Create a channel to the `OpenShell` server. /// /// When the endpoint uses `https://`, mTLS is configured using these env vars: /// - `OPENSHELL_TLS_CA` -- path to the CA certificate @@ -33,7 +34,11 @@ async fn connect_channel(endpoint: &str) -> Result { .connect_timeout(Duration::from_secs(10)) .http2_keep_alive_interval(Duration::from_secs(10)) .keep_alive_while_idle(true) - .keep_alive_timeout(Duration::from_secs(20)); + .keep_alive_timeout(Duration::from_secs(20)) + // Match the gateway-side HTTP/2 flow control (see `multiplex.rs`). + // Adaptive sizing lets idle streams stay tiny while bulk + // RelayStream data flows get a BDP-sized window. + .http2_adaptive_window(true); let tls_enabled = endpoint.starts_with("https://"); @@ -74,13 +79,62 @@ async fn connect_channel(endpoint: &str) -> Result { .wrap_err("failed to connect to OpenShell server") } -/// Connect to the OpenShell server (mTLS or plaintext based on endpoint scheme). -async fn connect(endpoint: &str) -> Result> { +/// Create a channel to the `OpenShell` server (public for use by `supervisor_session`). +pub async fn connect_channel_pub(endpoint: &str) -> Result { + connect_channel(endpoint).await +} + +/// Interceptor that injects the sandbox shared secret into every gRPC request. +/// +/// The server validates this header on sandbox-to-server RPCs (`GetSandboxConfig`, +/// `GetSandboxProviderEnvironment`, etc.) instead of requiring an OIDC Bearer token. +#[derive(Clone)] +pub struct SandboxSecretInterceptor { + secret: Option>, +} + +impl tonic::service::Interceptor for SandboxSecretInterceptor { + fn call( + &mut self, + mut req: tonic::Request<()>, + ) -> std::result::Result, tonic::Status> { + if let Some(ref val) = self.secret { + req.metadata_mut().insert("x-sandbox-secret", val.clone()); + } + Ok(req) + } +} + +type AuthenticatedClient = OpenShellClient>; +type AuthenticatedInferenceClient = + InferenceClient>; + +fn sandbox_secret_interceptor() -> SandboxSecretInterceptor { + let secret = std::env::var("OPENSHELL_SSH_HANDSHAKE_SECRET") + .ok() + .and_then(|s| s.parse().ok()); + SandboxSecretInterceptor { secret } +} + +/// Connect to the `OpenShell` server with sandbox secret authentication. +async fn connect(endpoint: &str) -> Result { let channel = connect_channel(endpoint).await?; - Ok(OpenShellClient::new(channel)) + Ok(OpenShellClient::with_interceptor( + channel, + sandbox_secret_interceptor(), + )) } -/// Fetch sandbox policy from OpenShell server via gRPC. +/// Connect to the inference service with sandbox secret authentication. +async fn connect_inference(endpoint: &str) -> Result { + let channel = connect_channel(endpoint).await?; + Ok(InferenceClient::with_interceptor( + channel, + sandbox_secret_interceptor(), + )) +} + +/// Fetch sandbox policy from `OpenShell` server via gRPC. /// /// Returns `Ok(Some(policy))` when the server has a policy configured, /// or `Ok(None)` when the sandbox was created without a policy (the sandbox @@ -97,7 +151,7 @@ pub async fn fetch_policy(endpoint: &str, sandbox_id: &str) -> Result, + client: &mut AuthenticatedClient, sandbox_id: &str, ) -> Result> { let response = client @@ -121,7 +175,7 @@ async fn fetch_policy_with_client( /// Sync a locally-discovered policy using an existing client connection. async fn sync_policy_with_client( - client: &mut OpenShellClient, + client: &mut AuthenticatedClient, sandbox: &str, policy: &ProtoSandboxPolicy, ) -> Result<()> { @@ -133,6 +187,7 @@ async fn sync_policy_with_client( setting_value: None, delete_setting: false, global: false, + merge_operations: vec![], }) .await .into_diagnostic() @@ -181,7 +236,7 @@ pub async fn sync_policy(endpoint: &str, sandbox: &str, policy: &ProtoSandboxPol sync_policy_with_client(&mut client, sandbox, policy).await } -/// Fetch provider environment variables for a sandbox from OpenShell server via gRPC. +/// Fetch provider environment variables for a sandbox from `OpenShell` server via gRPC. /// /// Returns a map of environment variable names to values derived from provider /// credentials configured on the sandbox. Returns an empty map if the sandbox @@ -204,13 +259,13 @@ pub async fn fetch_provider_environment( Ok(response.into_inner().environment) } -/// A reusable gRPC client for the OpenShell service. +/// A reusable gRPC client for the `OpenShell` service. /// /// Wraps a tonic channel connected once and reused for policy polling /// and status reporting, avoiding per-request TLS handshake overhead. #[derive(Clone)] pub struct CachedOpenShellClient { - client: OpenShellClient, + client: AuthenticatedClient, } /// Settings poll result returned by [`CachedOpenShellClient::poll_settings`]. @@ -221,7 +276,7 @@ pub struct SettingsPollResult { pub config_revision: u64, pub policy_source: PolicySource, /// Effective settings keyed by name. - pub settings: std::collections::HashMap, + pub settings: HashMap, /// When `policy_source` is `Global`, the version of the global policy revision. pub global_policy_version: u32, } @@ -229,13 +284,12 @@ pub struct SettingsPollResult { impl CachedOpenShellClient { pub async fn connect(endpoint: &str) -> Result { debug!(endpoint = %endpoint, "Connecting openshell gRPC client for policy polling"); - let channel = connect_channel(endpoint).await?; - let client = OpenShellClient::new(channel); + let client = connect(endpoint).await?; Ok(Self { client }) } /// Get a clone of the underlying tonic client for direct RPC calls. - pub fn raw_client(&self) -> OpenShellClient { + pub fn raw_client(&self) -> AuthenticatedClient { self.client.clone() } @@ -319,8 +373,7 @@ impl CachedOpenShellClient { pub async fn fetch_inference_bundle(endpoint: &str) -> Result { debug!(endpoint = %endpoint, "Fetching inference route bundle"); - let channel = connect_channel(endpoint).await?; - let mut client = InferenceClient::new(channel); + let mut client = connect_inference(endpoint).await?; let response = client .get_inference_bundle(GetInferenceBundleRequest {}) @@ -329,3 +382,32 @@ pub async fn fetch_inference_bundle(endpoint: &str) -> Result, + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, +} + +#[derive(Debug, Clone, Serialize, PartialEq, Eq)] +pub struct GraphqlOperationInfo { + pub operation_type: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub operation_name: Option, + pub fields: Vec, + pub persisted_query: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub persisted_query_hash: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub persisted_query_id: Option, +} + +pub struct GraphqlHttpRequest { + pub request: L7Request, + pub info: GraphqlRequestInfo, +} + +pub async fn parse_graphql_http_request( + client: &mut C, + max_body_bytes: usize, + canonicalize_options: crate::l7::path::CanonicalizeOptions, +) -> Result> { + let provider = crate::l7::rest::RestProvider::with_options(canonicalize_options); + let Some(mut request) = provider.parse_request(client).await? else { + return Ok(None); + }; + + let info = inspect_graphql_request(client, &mut request, max_body_bytes).await?; + + Ok(Some(GraphqlHttpRequest { request, info })) +} + +pub(crate) async fn inspect_graphql_request( + client: &mut C, + request: &mut L7Request, + max_body_bytes: usize, +) -> Result { + let header_str = header_str(request)?; + reject_unsupported_headers(header_str)?; + let body = read_body_for_inspection(client, request, max_body_bytes).await?; + Ok(classify_request(request, &body)) +} + +pub fn classify_request(request: &L7Request, body: &[u8]) -> GraphqlRequestInfo { + match classify_request_inner(request, body) { + Ok(operations) => GraphqlRequestInfo { + operations, + error: None, + }, + Err(err) => GraphqlRequestInfo { + operations: Vec::new(), + error: Some(err), + }, + } +} + +fn classify_request_inner( + request: &L7Request, + body: &[u8], +) -> std::result::Result, String> { + match request.action.to_ascii_uppercase().as_str() { + "GET" => classify_get(request), + "POST" => classify_post(body), + method => Err(format!("unsupported GraphQL HTTP method {method}")), + } +} + +fn classify_get(request: &L7Request) -> std::result::Result, String> { + let query = unique_query_value(&request.query_params, "query")?; + let operation_name = unique_query_value(&request.query_params, "operationName")?; + let extensions = unique_query_value(&request.query_params, "extensions")? + .and_then(|raw| serde_json::from_str::(&raw).ok()); + let id = unique_persisted_query_id(&request.query_params)?; + + classify_envelope( + query.as_deref(), + operation_name.as_deref(), + extensions.as_ref(), + id, + ) +} + +fn classify_post(body: &[u8]) -> std::result::Result, String> { + if body.is_empty() { + return Err("GraphQL POST body is empty".to_string()); + } + let value: Value = serde_json::from_slice(body) + .map_err(|err| format!("GraphQL request body is not valid JSON: {err}"))?; + + match value { + Value::Array(items) => { + if items.is_empty() { + return Err("GraphQL batch request is empty".to_string()); + } + let mut operations = Vec::new(); + for item in items { + operations.extend(classify_json_envelope(&item)?); + } + Ok(operations) + } + Value::Object(_) => classify_json_envelope(&value), + _ => Err("GraphQL JSON envelope must be an object or array".to_string()), + } +} + +fn classify_json_envelope(value: &Value) -> std::result::Result, String> { + let obj = value + .as_object() + .ok_or_else(|| "GraphQL batch item must be an object".to_string())?; + let query = obj.get("query").and_then(Value::as_str); + let operation_name = obj.get("operationName").and_then(Value::as_str); + let extensions = obj.get("extensions"); + let id = obj + .get("id") + .or_else(|| obj.get("documentId")) + .or_else(|| obj.get("queryId")) + .and_then(Value::as_str) + .map(ToString::to_string); + + classify_envelope(query, operation_name, extensions, id) +} + +fn classify_envelope( + query: Option<&str>, + operation_name: Option<&str>, + extensions: Option<&Value>, + persisted_id: Option, +) -> std::result::Result, String> { + let persisted_hash = persisted_query_hash(extensions); + let query = query.filter(|q| !q.trim().is_empty()); + + if let Some(query) = query { + let mut operation = classify_document(query, operation_name)?; + if let Some(hash) = persisted_hash { + operation.persisted_query = true; + operation.persisted_query_hash = Some(hash); + } + if let Some(id) = persisted_id { + operation.persisted_query = true; + operation.persisted_query_id = Some(id); + } + return Ok(vec![operation]); + } + + if persisted_hash.is_some() || persisted_id.is_some() { + return Ok(vec![GraphqlOperationInfo { + operation_type: String::new(), + operation_name: operation_name.map(ToString::to_string), + fields: Vec::new(), + persisted_query: true, + persisted_query_hash: persisted_hash, + persisted_query_id: persisted_id, + }]); + } + + Err("GraphQL request has no query document or persisted query identifier".to_string()) +} + +fn classify_document( + query: &str, + operation_name: Option<&str>, +) -> std::result::Result { + let parser = Parser::new(query).recursion_limit(128).token_limit(20_000); + let cst = parser.parse(); + let mut parse_errors = cst.errors(); + if let Some(err) = parse_errors.next() { + return Err(format!("GraphQL document parse error: {err}")); + } + + let document = cst.document(); + let mut operations = Vec::new(); + let mut fragments = HashMap::new(); + + for definition in document.definitions() { + match definition { + cst::Definition::OperationDefinition(operation) => operations.push(operation), + cst::Definition::FragmentDefinition(fragment) => { + if let Some(name) = fragment.fragment_name().and_then(|n| n.name()) { + fragments.insert(name.text().to_string(), fragment); + } + } + _ => {} + } + } + + if operations.is_empty() { + return Err("GraphQL document contains no executable operation".to_string()); + } + + let selected = if let Some(expected_name) = operation_name.filter(|name| !name.is_empty()) { + operations + .into_iter() + .find(|op| { + op.name() + .is_some_and(|name| name.text().as_ref() == expected_name) + }) + .ok_or_else(|| format!("GraphQL operationName {expected_name:?} was not found"))? + } else if operations.len() == 1 { + operations.remove(0) + } else { + return Err("GraphQL document has multiple operations but no operationName".to_string()); + }; + + let operation_type = operation_type(&selected); + let operation_name = selected.name().map(|name| name.text().to_string()); + let selection_set = selected + .selection_set() + .ok_or_else(|| "GraphQL operation has no selection set".to_string())?; + let mut fields = HashSet::new(); + let mut visited_fragments = HashSet::new(); + collect_root_fields( + selection_set, + &fragments, + &mut visited_fragments, + &mut fields, + ); + let mut fields: Vec<_> = fields.into_iter().collect(); + fields.sort(); + + Ok(GraphqlOperationInfo { + operation_type, + operation_name, + fields, + persisted_query: false, + persisted_query_hash: None, + persisted_query_id: None, + }) +} + +fn operation_type(operation: &cst::OperationDefinition) -> String { + let Some(operation_type) = operation.operation_type() else { + return "query".to_string(); + }; + if operation_type.mutation_token().is_some() { + "mutation".to_string() + } else if operation_type.subscription_token().is_some() { + "subscription".to_string() + } else { + "query".to_string() + } +} + +fn collect_root_fields( + selection_set: cst::SelectionSet, + fragments: &HashMap, + visited_fragments: &mut HashSet, + fields: &mut HashSet, +) { + for selection in selection_set.selections() { + match selection { + cst::Selection::Field(field) => { + if let Some(name) = field.name() { + fields.insert(name.text().to_string()); + } + } + cst::Selection::InlineFragment(fragment) => { + if let Some(selection_set) = fragment.selection_set() { + collect_root_fields(selection_set, fragments, visited_fragments, fields); + } + } + cst::Selection::FragmentSpread(spread) => { + let Some(name) = spread.fragment_name().and_then(|n| n.name()) else { + continue; + }; + let name = name.text().to_string(); + if !visited_fragments.insert(name.clone()) { + continue; + } + if let Some(fragment) = fragments.get(&name) + && let Some(selection_set) = fragment.selection_set() + { + collect_root_fields(selection_set, fragments, visited_fragments, fields); + } + } + } + } +} + +fn persisted_query_hash(extensions: Option<&Value>) -> Option { + extensions? + .get("persistedQuery")? + .get("sha256Hash")? + .as_str() + .filter(|hash| !hash.is_empty()) + .map(ToString::to_string) +} + +fn unique_query_value( + params: &HashMap>, + key: &str, +) -> std::result::Result, String> { + let Some(values) = params.get(key) else { + return Ok(None); + }; + if values.len() > 1 { + return Err(format!( + "GraphQL GET parameter {key:?} must not appear more than once" + )); + } + Ok(values.first().filter(|value| !value.is_empty()).cloned()) +} + +fn unique_persisted_query_id( + params: &HashMap>, +) -> std::result::Result, String> { + let mut selected: Option<(String, String)> = None; + for key in ["id", "documentId", "queryId"] { + let Some(value) = unique_query_value(params, key)? else { + continue; + }; + if let Some((existing_key, _)) = selected { + return Err(format!( + "GraphQL GET persisted-query id parameters {existing_key:?} and {key:?} must not be combined" + )); + } + selected = Some((key.to_string(), value)); + } + Ok(selected.map(|(_, value)| value)) +} + +async fn read_body_for_inspection( + client: &mut C, + request: &mut L7Request, + max_body_bytes: usize, +) -> Result> { + let header_end = request + .raw_header + .windows(4) + .position(|w| w == b"\r\n\r\n") + .map_or(request.raw_header.len(), |p| p + 4); + let overflow = request.raw_header[header_end..].to_vec(); + + match request.body_length { + BodyLength::None => Ok(Vec::new()), + BodyLength::ContentLength(len) => { + let len = usize::try_from(len) + .map_err(|_| miette!("GraphQL request body length exceeds platform limit"))?; + if len > max_body_bytes { + return Err(miette!( + "GraphQL request body exceeds {max_body_bytes} byte inspection limit" + )); + } + if overflow.len() > len { + return Err(miette!( + "GraphQL request contains more body bytes than Content-Length" + )); + } + let remaining = len - overflow.len(); + let mut body = overflow; + if remaining > 0 { + let start = body.len(); + body.resize(len, 0); + client + .read_exact(&mut body[start..]) + .await + .into_diagnostic()?; + } + request.raw_header.truncate(header_end); + request.raw_header.extend_from_slice(&body); + Ok(body) + } + BodyLength::Chunked => { + let body = read_chunked_body_for_inspection( + client, + request, + header_end, + overflow, + max_body_bytes, + ) + .await?; + normalize_chunked_request_to_content_length(request, header_end, &body)?; + Ok(body) + } + } +} + +fn normalize_chunked_request_to_content_length( + request: &mut L7Request, + header_end: usize, + body: &[u8], +) -> Result<()> { + let header_str = std::str::from_utf8(&request.raw_header[..header_end]) + .map_err(|_| miette!("GraphQL HTTP headers contain invalid UTF-8"))?; + let header_str = header_str + .strip_suffix("\r\n\r\n") + .ok_or_else(|| miette!("GraphQL HTTP headers missing terminator"))?; + + let mut normalized = Vec::with_capacity(header_str.len() + body.len() + 32); + for (idx, line) in header_str.split("\r\n").enumerate() { + if idx > 0 { + let name = line + .split_once(':') + .map(|(name, _)| name.trim().to_ascii_lowercase()); + if matches!( + name.as_deref(), + Some("transfer-encoding" | "content-length" | "trailer") + ) { + continue; + } + } + normalized.extend_from_slice(line.as_bytes()); + normalized.extend_from_slice(b"\r\n"); + } + normalized.extend_from_slice(format!("Content-Length: {}\r\n\r\n", body.len()).as_bytes()); + normalized.extend_from_slice(body); + + request.raw_header = normalized; + request.body_length = BodyLength::ContentLength(body.len() as u64); + Ok(()) +} + +async fn read_chunked_body_for_inspection( + client: &mut C, + request: &mut L7Request, + header_end: usize, + overflow: Vec, + max_body_bytes: usize, +) -> Result> { + let mut raw = overflow; + let mut decoded = Vec::new(); + let mut pos = 0usize; + + loop { + let size_line_end = loop { + if let Some(end) = find_crlf(&raw, pos) { + break end; + } + read_more(client, &mut raw, max_body_bytes).await?; + }; + let size_line = std::str::from_utf8(&raw[pos..size_line_end]) + .into_diagnostic() + .map_err(|_| miette!("Invalid UTF-8 in GraphQL chunk-size line"))?; + let size_token = size_line + .split(';') + .next() + .map(str::trim) + .unwrap_or_default(); + let chunk_size = usize::from_str_radix(size_token, 16) + .into_diagnostic() + .map_err(|_| miette!("Invalid GraphQL chunk size token: {size_token:?}"))?; + pos = size_line_end + 2; + + if decoded.len().saturating_add(chunk_size) > max_body_bytes { + return Err(miette!( + "GraphQL request body exceeds {max_body_bytes} byte inspection limit" + )); + } + + if chunk_size == 0 { + loop { + let trailer_end = loop { + if let Some(end) = find_crlf(&raw, pos) { + break end; + } + read_more(client, &mut raw, max_body_bytes).await?; + }; + let trailer_line = &raw[pos..trailer_end]; + pos = trailer_end + 2; + if trailer_line.is_empty() { + request.raw_header.truncate(header_end); + request.raw_header.extend_from_slice(&raw[..pos]); + return Ok(decoded); + } + } + } + + let chunk_end = pos + .checked_add(chunk_size) + .ok_or_else(|| miette!("GraphQL chunk size overflow"))?; + let chunk_with_crlf_end = chunk_end + .checked_add(2) + .ok_or_else(|| miette!("GraphQL chunk size overflow"))?; + while raw.len() < chunk_with_crlf_end { + read_more(client, &mut raw, max_body_bytes).await?; + } + decoded.extend_from_slice(&raw[pos..chunk_end]); + if raw.get(chunk_end..chunk_with_crlf_end) != Some(&b"\r\n"[..]) { + return Err(miette!("GraphQL chunk payload missing terminating CRLF")); + } + pos = chunk_with_crlf_end; + } +} + +async fn read_more( + client: &mut C, + raw: &mut Vec, + max_body_bytes: usize, +) -> Result<()> { + if raw.len() > max_body_bytes.saturating_mul(2).max(max_body_bytes) { + return Err(miette!( + "GraphQL chunked request body exceeds inspection framing limit" + )); + } + let mut buf = [0u8; 8192]; + let n = client.read(&mut buf).await.into_diagnostic()?; + if n == 0 { + return Err(miette!("GraphQL chunked body ended before terminator")); + } + raw.extend_from_slice(&buf[..n]); + Ok(()) +} + +fn find_crlf(buf: &[u8], start: usize) -> Option { + buf.get(start..)? + .windows(2) + .position(|w| w == b"\r\n") + .map(|p| start + p) +} + +fn header_str(request: &L7Request) -> Result<&str> { + let header_end = request + .raw_header + .windows(4) + .position(|w| w == b"\r\n\r\n") + .map_or(request.raw_header.len(), |p| p + 4); + std::str::from_utf8(&request.raw_header[..header_end]) + .map_err(|_| miette!("GraphQL HTTP headers contain invalid UTF-8")) +} + +fn reject_unsupported_headers(headers: &str) -> Result<()> { + for line in headers.lines().skip(1) { + let lower = line.to_ascii_lowercase(); + if lower.starts_with("content-encoding:") { + let encoding = lower.split_once(':').map_or("", |(_, v)| v.trim()); + if !encoding.is_empty() && encoding != "identity" { + return Err(miette!( + "GraphQL request content-encoding {encoding:?} is not supported" + )); + } + } + if lower.starts_with("content-type:") { + let content_type = lower.split_once(':').map_or("", |(_, v)| v.trim()); + if content_type.starts_with("multipart/") { + return Err(miette!("GraphQL multipart requests are not supported")); + } + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn request(method: &str, target: &str) -> L7Request { + L7Request { + action: method.to_string(), + target: target.to_string(), + query_params: crate::l7::rest::parse_target_query(target).unwrap().1, + raw_header: format!("{method} {target} HTTP/1.1\r\nHost: example.com\r\n\r\n") + .into_bytes(), + body_length: BodyLength::None, + } + } + + #[test] + fn classifies_simple_query() { + let req = request("POST", "/graphql"); + let info = classify_request(&req, br#"{"query":"query Viewer { viewer { login } }"}"#); + assert_eq!(info.error, None); + assert_eq!(info.operations[0].operation_type, "query"); + assert_eq!(info.operations[0].fields, vec!["viewer"]); + } + + #[test] + fn classifies_mutation_field_not_alias() { + let req = request("POST", "/graphql"); + let info = classify_request( + &req, + br#"{"query":"mutation M { safeAlias: volumeDelete(volumeId:\"x\") { id } }","operationName":"M"}"#, + ); + assert_eq!(info.error, None); + assert_eq!(info.operations[0].operation_type, "mutation"); + assert_eq!(info.operations[0].operation_name.as_deref(), Some("M")); + assert_eq!(info.operations[0].fields, vec!["volumeDelete"]); + } + + #[test] + fn expands_root_fragments() { + let req = request("POST", "/graphql"); + let info = classify_request( + &req, + br#"{"query":"query Q { ...RootFields } fragment RootFields on Query { viewer repository(owner:\"o\", name:\"r\") { id } }"}"#, + ); + assert_eq!(info.error, None); + assert_eq!(info.operations[0].fields, vec!["repository", "viewer"]); + } + + #[test] + fn multiple_operations_without_name_errors() { + let req = request("POST", "/graphql"); + let info = classify_request( + &req, + br#"{"query":"query A { viewer { login } } query B { rateLimit { limit } }"}"#, + ); + assert!(info.error.unwrap().contains("multiple operations")); + } + + #[test] + fn detects_hash_only_apollo_persisted_query() { + let req = request("POST", "/graphql"); + let info = classify_request( + &req, + br#"{"operationName":"Viewer","extensions":{"persistedQuery":{"version":1,"sha256Hash":"abc123"}}}"#, + ); + assert_eq!(info.error, None); + let op = &info.operations[0]; + assert!(op.persisted_query); + assert_eq!(op.operation_name.as_deref(), Some("Viewer")); + assert_eq!(op.persisted_query_hash.as_deref(), Some("abc123")); + } + + #[test] + fn graphql_get_rejects_duplicate_query_parameter() { + let req = request( + "GET", + "/graphql?query=query+Viewer+%7B+viewer+%7B+login+%7D+%7D&query=mutation+Delete+%7B+volumeDelete(volumeId%3A%22x%22)+%7B+id+%7D+%7D", + ); + let info = classify_request(&req, b""); + assert!( + info.error + .as_deref() + .is_some_and(|err| err.contains("must not appear more than once")), + "expected duplicate control parameter error, got {info:?}" + ); + } + + #[test] + fn graphql_get_rejects_ambiguous_persisted_query_ids() { + let req = request("GET", "/graphql?id=one&queryId=two"); + let info = classify_request(&req, b""); + assert!( + info.error + .as_deref() + .is_some_and(|err| err.contains("must not be combined")), + "expected ambiguous persisted-query id error, got {info:?}" + ); + } + + #[tokio::test] + async fn chunked_graphql_post_is_normalized_after_inspection() { + let body = br#"{"query":"query Viewer { viewer { login } }"}"#; + let mut raw_header = + b"POST /graphql HTTP/1.1\r\nHost: example.com\r\nTransfer-Encoding: chunked\r\nTrailer: X-Sig\r\nX-Test: yes\r\n\r\n" + .to_vec(); + raw_header.extend_from_slice(format!("{:x}\r\n", body.len()).as_bytes()); + raw_header.extend_from_slice(body); + raw_header.extend_from_slice(b"\r\n0\r\nX-Sig: ignored\r\n\r\n"); + + let mut req = L7Request { + action: "POST".to_string(), + target: "/graphql".to_string(), + query_params: HashMap::new(), + raw_header, + body_length: BodyLength::Chunked, + }; + let mut client = tokio::io::empty(); + + let info = inspect_graphql_request(&mut client, &mut req, DEFAULT_MAX_BODY_BYTES) + .await + .expect("chunked body should inspect"); + + assert_eq!(info.error, None); + assert!(matches!( + req.body_length, + BodyLength::ContentLength(len) if len == body.len() as u64 + )); + let forwarded = String::from_utf8_lossy(&req.raw_header); + assert!(forwarded.contains(&format!("Content-Length: {}", body.len()))); + assert!(forwarded.contains("X-Test: yes\r\n")); + assert!(!forwarded.to_ascii_lowercase().contains("transfer-encoding")); + assert!(!forwarded.to_ascii_lowercase().contains("trailer:")); + assert!(req.raw_header.ends_with(body)); + } + + #[tokio::test] + async fn absolute_form_chunked_graphql_post_classifies_after_inspection() { + let body = br#"{"query":"query Viewer { viewer { login } }"}"#; + let mut raw_header = + b"POST http://example.com/graphql HTTP/1.1\r\nHost: example.com\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\nConnection: close\r\n\r\n" + .to_vec(); + raw_header.extend_from_slice(format!("{:x}\r\n", body.len()).as_bytes()); + raw_header.extend_from_slice(body); + raw_header.extend_from_slice(b"\r\n0\r\n\r\n"); + + let mut req = L7Request { + action: "POST".to_string(), + target: "/graphql".to_string(), + query_params: HashMap::new(), + raw_header, + body_length: BodyLength::Chunked, + }; + let mut client = tokio::io::empty(); + + let info = inspect_graphql_request(&mut client, &mut req, DEFAULT_MAX_BODY_BYTES) + .await + .expect("absolute-form chunked body should inspect"); + + assert_eq!(info.error, None); + assert_eq!(info.operations[0].operation_type, "query"); + assert_eq!(info.operations[0].fields, vec!["viewer"]); + } + + #[tokio::test] + async fn absolute_form_chunked_graphql_post_is_allowed_by_field_policy() { + let body = br#"{"query":"query Viewer { viewer { login } }"}"#; + let mut raw_header = + b"POST http://host.openshell.internal:8080/graphql HTTP/1.1\r\nHost: host.openshell.internal:8080\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\nConnection: close\r\n\r\n" + .to_vec(); + raw_header.extend_from_slice(format!("{:x}\r\n", body.len()).as_bytes()); + raw_header.extend_from_slice(body); + raw_header.extend_from_slice(b"\r\n0\r\n\r\n"); + + let mut req = L7Request { + action: "POST".to_string(), + target: "/graphql".to_string(), + query_params: HashMap::new(), + raw_header, + body_length: BodyLength::Chunked, + }; + let mut client = tokio::io::empty(); + let info = inspect_graphql_request(&mut client, &mut req, DEFAULT_MAX_BODY_BYTES) + .await + .expect("chunked body should inspect"); + + let data = r" +network_policies: + test_graphql_l7: + name: test_graphql_l7 + endpoints: + - host: host.openshell.internal + port: 8080 + protocol: graphql + enforcement: enforce + persisted_queries: allow_registered + graphql_persisted_queries: + abc123: + operation_type: query + operation_name: Viewer + fields: [viewer] + rules: + - allow: + operation_type: query + fields: [viewer] + - allow: + operation_type: mutation + fields: [createIssue] + deny_rules: + - operation_type: mutation + fields: [deleteRepository] + binaries: + - { path: /usr/bin/python3 } +"; + let engine = crate::opa::OpaEngine::from_strings( + include_str!("../../data/sandbox-policy.rego"), + data, + ) + .expect("policy should load"); + let ctx = crate::l7::relay::L7EvalContext { + host: "host.openshell.internal".to_string(), + port: 8080, + policy_name: "test_graphql_l7".to_string(), + binary_path: "/usr/bin/python3".to_string(), + ancestors: Vec::new(), + cmdline_paths: Vec::new(), + secret_resolver: None, + }; + let request_info = crate::l7::L7RequestInfo { + action: req.action, + target: req.target, + query_params: req.query_params, + graphql: Some(info), + }; + + let tunnel_engine = engine + .clone_engine_for_tunnel(engine.current_generation()) + .expect("tunnel engine should clone"); + let (allowed, reason) = + crate::l7::relay::evaluate_l7_request(&tunnel_engine, &ctx, &request_info) + .expect("evaluation should complete"); + + assert!(allowed, "expected query to be allowed, got {reason}"); + } +} diff --git a/crates/openshell-sandbox/src/l7/mod.rs b/crates/openshell-sandbox/src/l7/mod.rs index d354f562b2..5301ac4d56 100644 --- a/crates/openshell-sandbox/src/l7/mod.rs +++ b/crates/openshell-sandbox/src/l7/mod.rs @@ -8,7 +8,9 @@ //! doing a raw `copy_bidirectional`. Each request within the tunnel is parsed, //! evaluated against OPA policy, and either forwarded or denied. +pub mod graphql; pub mod inference; +pub mod path; pub mod provider; pub mod relay; pub mod rest; @@ -18,6 +20,7 @@ pub mod tls; #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum L7Protocol { Rest, + Graphql, Sql, } @@ -25,6 +28,7 @@ impl L7Protocol { pub fn parse(s: &str) -> Option { match s.to_ascii_lowercase().as_str() { "rest" => Some(Self::Rest), + "graphql" => Some(Self::Graphql), "sql" => Some(Self::Sql), _ => None, } @@ -57,8 +61,17 @@ pub enum EnforcementMode { #[derive(Debug, Clone)] pub struct L7EndpointConfig { pub protocol: L7Protocol, + /// Optional endpoint-level HTTP path glob used to select between L7 + /// protocols that share the same host:port. + pub path: String, pub tls: TlsMode, pub enforcement: EnforcementMode, + /// Maximum GraphQL request body bytes to buffer for inspection. + pub graphql_max_body_bytes: usize, + /// When true, percent-encoded `/` (`%2F`) is preserved in path segments + /// rather than rejected at the parser. Needed by upstreams like GitLab + /// that embed `%2F` in namespaced project paths. Defaults to false. + pub allow_encoded_slash: bool, } /// Result of an L7 policy decision for a single request. @@ -78,6 +91,8 @@ pub struct L7RequestInfo { pub target: String, /// Decoded query parameter multimap for REST requests. pub query_params: std::collections::HashMap>, + /// Parsed GraphQL operation metadata for GraphQL endpoints. + pub graphql: Option, } /// Parse an L7 endpoint config from a regorus Value (returned by Rego query). @@ -122,13 +137,49 @@ pub fn parse_l7_config(val: ®orus::Value) -> Option { _ => EnforcementMode::Audit, }; + let allow_encoded_slash = get_object_bool(val, "allow_encoded_slash").unwrap_or(false); + let graphql_max_body_bytes = get_object_u64(val, "graphql_max_body_bytes") + .and_then(|v| usize::try_from(v).ok()) + .filter(|v| *v > 0) + .unwrap_or(graphql::DEFAULT_MAX_BODY_BYTES); + Some(L7EndpointConfig { protocol, + path: get_object_str(val, "path").unwrap_or_default(), tls, enforcement, + graphql_max_body_bytes, + allow_encoded_slash, }) } +impl L7EndpointConfig { + pub fn matches_path(&self, path: &str) -> bool { + endpoint_path_matches(&self.path, path) + } + + pub fn path_specificity(&self) -> usize { + if self.path.is_empty() { + 0 + } else { + self.path.chars().filter(|c| *c != '*').count() + } + } +} + +pub fn endpoint_path_matches(pattern: &str, path: &str) -> bool { + if pattern.is_empty() || pattern == "**" || pattern == "/**" { + return true; + } + if pattern == path { + return true; + } + if let Some(prefix) = pattern.strip_suffix("/**") { + return path == prefix || path.starts_with(&format!("{prefix}/")); + } + glob::Pattern::new(pattern).is_ok_and(|glob| glob.matches(path)) +} + /// Parse the `tls` field from an endpoint config, independent of L7 protocol. /// /// Used to check for `tls: skip` even on L4-only endpoints (no `protocol` @@ -136,11 +187,35 @@ pub fn parse_l7_config(val: ®orus::Value) -> Option { pub fn parse_tls_mode(val: ®orus::Value) -> TlsMode { match get_object_str(val, "tls").as_deref() { Some("skip") => TlsMode::Skip, - Some("terminate") | Some("passthrough") => TlsMode::Auto, // deprecation logged by parse_l7_config + // "terminate" and "passthrough" are deprecated aliases (logged by parse_l7_config); fall through to Auto. _ => TlsMode::Auto, } } +/// Extract a bool value from a regorus object. Returns `None` when the key +/// is absent or not a boolean. +fn get_object_bool(val: ®orus::Value, key: &str) -> Option { + let key_val = regorus::Value::String(key.into()); + match val { + regorus::Value::Object(map) => match map.get(&key_val) { + Some(regorus::Value::Bool(b)) => Some(*b), + _ => None, + }, + _ => None, + } +} + +fn get_object_u64(val: ®orus::Value, key: &str) -> Option { + let key_val = regorus::Value::String(key.into()); + match val { + regorus::Value::Object(map) => match map.get(&key_val) { + Some(regorus::Value::Number(n)) => n.as_u64(), + _ => None, + }, + _ => None, + } +} + /// Extract a string value from a regorus object. fn get_object_str(val: ®orus::Value, key: &str) -> Option { let key_val = regorus::Value::String(key.into()); @@ -199,6 +274,85 @@ fn check_glob_syntax(pattern: &str) -> Option { None } +fn validate_graphql_operation_type( + errors: &mut Vec, + loc: &str, + value: Option<&str>, + required: bool, +) { + let Some(value) = value.filter(|v| !v.is_empty()) else { + if required { + errors.push(format!( + "{loc}.operation_type: required for GraphQL L7 rules" + )); + } + return; + }; + + let valid = ["query", "mutation", "subscription", "*"]; + if !valid.contains(&value.to_ascii_lowercase().as_str()) { + errors.push(format!( + "{loc}.operation_type: expected query, mutation, subscription, or *, got '{value}'" + )); + } +} + +fn validate_graphql_fields( + errors: &mut Vec, + warnings: &mut Vec, + loc: &str, + fields: Option<&serde_json::Value>, +) { + let Some(fields) = fields else { + return; + }; + let Some(items) = fields.as_array() else { + errors.push(format!( + "{loc}.fields: expected array of GraphQL root field globs" + )); + return; + }; + if items.is_empty() { + errors.push(format!( + "{loc}.fields: list must not be empty; omit fields to match all root fields" + )); + return; + } + for item in items { + let Some(field) = item.as_str() else { + errors.push(format!("{loc}.fields: all values must be strings")); + continue; + }; + if field.is_empty() { + errors.push(format!("{loc}.fields: field glob must not be empty")); + } else if let Some(warning) = check_glob_syntax(field) { + warnings.push(format!("{loc}.fields: {warning}")); + } + } +} + +fn validate_graphql_rule( + errors: &mut Vec, + warnings: &mut Vec, + loc: &str, + rule: &serde_json::Value, + required: bool, +) { + validate_graphql_operation_type( + errors, + loc, + rule.get("operation_type").and_then(|v| v.as_str()), + required, + ); + if let Some(name) = rule.get("operation_name").and_then(|v| v.as_str()) + && !name.is_empty() + && let Some(warning) = check_glob_syntax(name) + { + warnings.push(format!("{loc}.operation_name: {warning}")); + } + validate_graphql_fields(errors, warnings, loc, rule.get("fields")); +} + /// Validate L7 policy configuration in the loaded OPA data. /// /// Returns a list of errors and warnings. Errors should prevent sandbox startup; @@ -229,21 +383,32 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< .and_then(|v| v.as_array()) .is_some_and(|a| !a.is_empty()); let host = ep.get("host").and_then(|v| v.as_str()).unwrap_or(""); + let endpoint_path = ep.get("path").and_then(|v| v.as_str()).unwrap_or(""); // Read ports from either "ports" array or scalar "port". - let ports: Vec = ep - .get("ports") - .and_then(|v| v.as_array()) - .map(|arr| arr.iter().filter_map(|v| v.as_u64()).collect()) - .unwrap_or_else(|| { + let ports: Vec = ep.get("ports").and_then(|v| v.as_array()).map_or_else( + || { ep.get("port") .and_then(serde_json::Value::as_u64) .filter(|p| *p > 0) .into_iter() .collect() - }); + }, + |arr| arr.iter().filter_map(serde_json::Value::as_u64).collect(), + ); let loc = format!("{name}.endpoints[{i}]"); + if !endpoint_path.is_empty() { + if !endpoint_path.starts_with('/') && endpoint_path != "**" { + errors.push(format!( + "{loc}: endpoint path must start with '/' or be '**', got '{endpoint_path}'" + )); + } + if let Some(warning) = check_glob_syntax(endpoint_path) { + warnings.push(format!("{loc}.path: {warning}")); + } + } + // Validate host wildcard patterns. if host.contains('*') { if host == "*" || host == "**" { @@ -295,6 +460,57 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< )); } + if !protocol.is_empty() && L7Protocol::parse(protocol).is_none() { + errors.push(format!( + "{loc}: unknown protocol '{protocol}' (expected rest, graphql, or sql)" + )); + } + + if let Some(mode) = ep.get("persisted_queries").and_then(|v| v.as_str()) + && !mode.is_empty() + && mode != "deny" + && mode != "allow_registered" + { + errors.push(format!( + "{loc}: persisted_queries must be 'deny' or 'allow_registered', got '{mode}'" + )); + } + + if ep.get("graphql_max_body_bytes").is_some() { + let valid_max = ep + .get("graphql_max_body_bytes") + .and_then(serde_json::Value::as_u64) + .is_some_and(|v| v > 0); + if !valid_max { + errors.push(format!( + "{loc}: graphql_max_body_bytes must be a positive integer" + )); + } + } + + if protocol != "graphql" + && (ep.get("persisted_queries").is_some() + || ep.get("graphql_persisted_queries").is_some() + || ep.get("graphql_max_body_bytes").is_some()) + { + warnings.push(format!( + "{loc}: GraphQL-specific endpoint fields are ignored unless protocol is graphql" + )); + } + + if let Some(registry_value) = ep.get("graphql_persisted_queries") { + let Some(registry) = registry_value.as_object() else { + errors.push(format!( + "{loc}: graphql_persisted_queries must be a map keyed by hash or saved-query id" + )); + continue; + }; + for (key, op) in registry { + let registry_loc = format!("{loc}.graphql_persisted_queries[{key}]"); + validate_graphql_rule(&mut errors, &mut warnings, ®istry_loc, op, true); + } + } + // Deprecated tls values: warn but don't error if tls == "terminate" || tls == "passthrough" { warnings.push(format!( @@ -371,10 +587,10 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< } // Validate path glob syntax - if let Some(path) = deny_rule.get("path").and_then(|p| p.as_str()) { - if let Some(warning) = check_glob_syntax(path) { - warnings.push(format!("{deny_loc}.path: {warning}")); - } + if let Some(path) = deny_rule.get("path").and_then(|p| p.as_str()) + && let Some(warning) = check_glob_syntax(path) + { + warnings.push(format!("{deny_loc}.path: {warning}")); } // Validate query matchers — mirrors allow-side validation exactly @@ -477,12 +693,29 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< } // SQL command validation - if let Some(command) = deny_rule.get("command").and_then(|c| c.as_str()) { - if !command.is_empty() && protocol == "rest" { - warnings.push(format!( - "{deny_loc}: command is for SQL protocol, not REST" - )); - } + if let Some(command) = deny_rule.get("command").and_then(|c| c.as_str()) + && !command.is_empty() + && protocol == "rest" + { + warnings + .push(format!("{deny_loc}: command is for SQL protocol, not REST")); + } + + if protocol == "graphql" { + validate_graphql_rule( + &mut errors, + &mut warnings, + &deny_loc, + deny_rule, + true, + ); + } else if deny_rule.get("operation_type").is_some() + || deny_rule.get("operation_name").is_some() + || deny_rule.get("fields").is_some() + { + warnings.push(format!( + "{deny_loc}: GraphQL rule fields are ignored unless protocol is graphql" + )); } } } @@ -624,6 +857,17 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< } } } + + if has_rules + && protocol == "graphql" + && let Some(rules) = ep.get("rules").and_then(|v| v.as_array()) + { + for (rule_idx, rule) in rules.iter().enumerate() { + let allow = rule.get("allow").unwrap_or(rule); + let rule_loc = format!("{loc}.rules[{rule_idx}].allow"); + validate_graphql_rule(&mut errors, &mut warnings, &rule_loc, allow, true); + } + } } } @@ -666,22 +910,35 @@ pub fn expand_access_presets(data: &mut serde_json::Value) { continue; } - let rules = match access.as_str() { - "read-only" => vec![ - rule_json("GET", "**"), - rule_json("HEAD", "**"), - rule_json("OPTIONS", "**"), - ], - "read-write" => vec![ - rule_json("GET", "**"), - rule_json("HEAD", "**"), - rule_json("OPTIONS", "**"), - rule_json("POST", "**"), - rule_json("PUT", "**"), - rule_json("PATCH", "**"), - ], - "full" => vec![rule_json("*", "**")], - _ => continue, + let protocol = ep + .get("protocol") + .and_then(|v| v.as_str()) + .unwrap_or("rest"); + let rules = if protocol == "graphql" { + match access.as_str() { + "read-only" => vec![graphql_rule_json("query")], + "read-write" => vec![graphql_rule_json("query"), graphql_rule_json("mutation")], + "full" => vec![graphql_rule_json("*")], + _ => continue, + } + } else { + match access.as_str() { + "read-only" => vec![ + rule_json("GET", "**"), + rule_json("HEAD", "**"), + rule_json("OPTIONS", "**"), + ], + "read-write" => vec![ + rule_json("GET", "**"), + rule_json("HEAD", "**"), + rule_json("OPTIONS", "**"), + rule_json("POST", "**"), + rule_json("PUT", "**"), + rule_json("PATCH", "**"), + ], + "full" => vec![rule_json("*", "**")], + _ => continue, + } }; ep.as_object_mut() @@ -700,6 +957,14 @@ fn rule_json(method: &str, path: &str) -> serde_json::Value { }) } +fn graphql_rule_json(operation_type: &str) -> serde_json::Value { + serde_json::json!({ + "allow": { + "operation_type": operation_type + } + }) +} + #[cfg(test)] mod tests { use super::*; @@ -746,6 +1011,26 @@ mod tests { assert!(parse_l7_config(&val).is_none()); } + #[test] + fn parse_l7_config_allow_encoded_slash_defaults_false() { + let val = regorus::Value::from_json_str( + r#"{"protocol": "rest", "host": "api.example.com", "port": 443}"#, + ) + .unwrap(); + let config = parse_l7_config(&val).unwrap(); + assert!(!config.allow_encoded_slash); + } + + #[test] + fn parse_l7_config_allow_encoded_slash_opt_in() { + let val = regorus::Value::from_json_str( + r#"{"protocol": "rest", "host": "gitlab.example.com", "port": 443, "allow_encoded_slash": true}"#, + ) + .unwrap(); + let config = parse_l7_config(&val).unwrap(); + assert!(config.allow_encoded_slash); + } + #[test] fn validate_rules_and_access_mutual_exclusion() { let data = serde_json::json!({ @@ -936,6 +1221,81 @@ mod tests { assert_eq!(rules[0]["allow"]["path"].as_str().unwrap(), "**"); } + #[test] + fn expand_graphql_readonly_preset() { + let mut data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "api.example.com", + "port": 443, + "protocol": "graphql", + "access": "read-only" + }], + "binaries": [] + } + } + }); + expand_access_presets(&mut data); + let rules = data["network_policies"]["test"]["endpoints"][0]["rules"] + .as_array() + .unwrap(); + assert_eq!(rules.len(), 1); + assert_eq!( + rules[0]["allow"]["operation_type"].as_str().unwrap(), + "query" + ); + } + + #[test] + fn validate_graphql_rule_requires_operation_type() { + let data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "api.example.com", + "port": 443, + "protocol": "graphql", + "rules": [{ + "allow": { + "fields": ["viewer"] + } + }] + }], + "binaries": [] + } + } + }); + let (errors, _warnings) = validate_l7_policies(&data); + assert!( + errors.iter().any(|e| e.contains("operation_type")), + "GraphQL rules should require operation_type: {errors:?}" + ); + } + + #[test] + fn validate_graphql_persisted_query_mode() { + let data = serde_json::json!({ + "network_policies": { + "test": { + "endpoints": [{ + "host": "api.example.com", + "port": 443, + "protocol": "graphql", + "access": "full", + "persisted_queries": "allow_all" + }], + "binaries": [] + } + } + }); + let (errors, _warnings) = validate_l7_policies(&data); + assert!( + errors.iter().any(|e| e.contains("persisted_queries")), + "invalid persisted query mode should be rejected: {errors:?}" + ); + } + #[test] fn l4_only_endpoint_untouched() { let mut data = serde_json::json!({ diff --git a/crates/openshell-sandbox/src/l7/path.rs b/crates/openshell-sandbox/src/l7/path.rs new file mode 100644 index 0000000000..db2e4e9847 --- /dev/null +++ b/crates/openshell-sandbox/src/l7/path.rs @@ -0,0 +1,560 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! HTTP request-target canonicalization for L7 policy enforcement. +//! +//! The L7 REST proxy evaluates OPA rules against the request path and +//! forwards the raw request line to the upstream server. If the path the +//! policy sees is not the path the upstream dispatches on, any path-based +//! allow rule can be bypassed with non-canonical encodings (`..`, `%2e%2e`, +//! `//`, `;params`). This module resolves that divergence by producing a +//! single canonical path that is both the input to policy evaluation and +//! the bytes written onto the wire. +//! +//! Behavior for v1: +//! - Percent-decode unreserved path bytes; preserve the rest as uppercase +//! `%HH`. +//! - Resolve `.` and `..` segments per RFC 3986 Section 5.2.4. `..` that +//! would escape the root is rejected rather than silently clamped to +//! `/` — non-canonical input is almost always adversarial. +//! - Collapse repeated slashes. +//! - Reject control bytes (`0x00..=0x1F`, `0x7F`), fragments in the +//! request-target, raw non-ASCII bytes, and paths that cannot be parsed +//! as origin-form. +//! - Strip trailing `;params` from each segment by default (Tomcat-class +//! `;jsessionid` ACL-bypass mitigation). +//! - Reject `%2F` (encoded slash) inside a segment by default. Operators +//! can opt in per-endpoint for APIs that rely on encoded slashes in +//! slugs. + +use thiserror::Error; + +/// Reasons a request-target can be rejected at the canonicalization boundary. +#[derive(Debug, Clone, PartialEq, Eq, Error)] +pub enum CanonicalizeError { + #[error("request-target contains a null or control byte")] + NullOrControlByte, + #[error("request-target contains an invalid percent-encoded sequence")] + InvalidPercentEncoding, + #[error("request-target contains an encoded '/' (%2F) which is not allowed on this endpoint")] + EncodedSlashNotAllowed, + #[error("request-target contains a fragment")] + FragmentInRequestTarget, + #[error("request-target contains raw non-ASCII bytes; non-ASCII must be percent-encoded")] + NonAscii, + #[error("request-target's `..` segment would escape the path root")] + TraversalAboveRoot, + #[error("request-target exceeds the configured maximum length")] + PathTooLong, + #[error("request-target is not a valid origin-form path")] + MalformedTarget, +} + +/// Options controlling canonicalization strictness. +/// +/// Produced by the endpoint configuration. Defaults are intentionally strict: +/// operators opt in to looser behavior per-endpoint when the upstream API +/// requires it. +#[derive(Debug, Clone, Copy)] +pub struct CanonicalizeOptions { + /// When `true`, `%2F` inside a segment is preserved (re-emitted as + /// `%2F` on the wire) rather than rejected. Defaults to `false`. + pub allow_encoded_slash: bool, + /// When `true`, RFC 3986 path parameters (`;param`) are stripped from + /// each segment before policy evaluation and before forwarding. + /// Defaults to `true`: path parameters are an ambiguity surface + /// historically used to bypass ACLs and are not part of any policy + /// we author. + pub strip_path_parameters: bool, +} + +impl Default for CanonicalizeOptions { + fn default() -> Self { + Self { + allow_encoded_slash: false, + strip_path_parameters: true, + } + } +} + +/// Result of a successful canonicalization. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CanonicalPath { + /// The canonical path. Always starts with `/`. Contains no `.`/`..` + /// segments, no doubled slashes, and no `;params` (when stripping is + /// enabled). + pub path: String, + /// `true` if the canonical form differs from the input. Callers use + /// this to decide whether to rewrite the outbound request line. + pub rewritten: bool, +} + +/// Maximum accepted length of an origin-form path (bytes). +pub(crate) const MAX_PATH_LEN: usize = 4 * 1024; + +/// Sentinel byte used to represent a `%2F`-decoded slash inside a segment. +/// Chosen from the C0 control range so no legitimate decoded byte collides +/// with it; any raw `0x01` in the input is rejected up front. +const ENCODED_SLASH_SENTINEL: u8 = 0x01; + +/// Canonicalize an HTTP request-target's path component. +/// +/// Accepts origin-form (`"/a/b?q=1"`) or absolute-form (`"http://h/a/b"`) +/// targets. Asterisk-form (`"*"`, used only for `OPTIONS *`) is rejected +/// because the L7 enforcement pipeline does not handle it. +/// +/// Returns the canonical path plus the original query suffix (byte-for-byte +/// as supplied by the client). Query-parameter parsing is left to the +/// caller — this function only operates on the path component. +pub fn canonicalize_request_target( + target: &str, + opts: &CanonicalizeOptions, +) -> Result<(CanonicalPath, Option), CanonicalizeError> { + // 1. Reject control bytes and raw non-ASCII outright. These tests also + // cover CR/LF which are never legal in a single-line request-target. + for &b in target.as_bytes() { + if b == 0 || b == b'\n' || b == b'\r' || b == b'\t' || b == 0x7F { + return Err(CanonicalizeError::NullOrControlByte); + } + if b < 0x20 { + return Err(CanonicalizeError::NullOrControlByte); + } + if b >= 0x80 { + return Err(CanonicalizeError::NonAscii); + } + } + + // 2. Reject fragments — forbidden in request-target per RFC 7230. + if target.contains('#') { + return Err(CanonicalizeError::FragmentInRequestTarget); + } + + // 3. Split off query at the first `?`. Query is preserved verbatim. + let (path_part, query_part) = match target.split_once('?') { + Some((p, q)) => (p, Some(q.to_string())), + None => (target, None), + }; + + // 4. Handle absolute-form by stripping scheme://authority. + let raw_path = path_part.find("://").map_or(path_part, |idx| { + let after_scheme = &path_part[idx + 3..]; + after_scheme + .find('/') + .map_or("/", |slash| &after_scheme[slash..]) + }); + + // 5. Empty is equivalent to "/". + let raw_path = if raw_path.is_empty() { "/" } else { raw_path }; + + // 6. Must begin with '/' (origin-form). + if !raw_path.starts_with('/') { + return Err(CanonicalizeError::MalformedTarget); + } + + // 7. Length bound. + if raw_path.len() > MAX_PATH_LEN { + return Err(CanonicalizeError::PathTooLong); + } + + // 8. Percent-decode the path into bytes. `%2F` is replaced by a + // sentinel byte so that subsequent `/` splitting cannot confuse it + // with a real path separator. + let decoded = percent_decode_with_sentinel(raw_path.as_bytes(), opts.allow_encoded_slash)?; + + // 9. Split on literal `/` and resolve dot-segments. + let segments = split_path_segments(&decoded); + let resolved = resolve_dot_segments(segments)?; + + // 10. Reconstruct. Strip `;params` per segment if requested; re-encode + // any byte that must be percent-encoded in the pchar set. + let canonical = build_canonical_path(&resolved, decoded.last().copied() == Some(b'/'), *opts); + + let rewritten = canonical != raw_path; + Ok(( + CanonicalPath { + path: canonical, + rewritten, + }, + query_part, + )) +} + +// --------------------------------------------------------------------------- +// Internals +// --------------------------------------------------------------------------- + +fn percent_decode_with_sentinel( + raw: &[u8], + allow_encoded_slash: bool, +) -> Result, CanonicalizeError> { + let mut out = Vec::with_capacity(raw.len()); + let mut i = 0; + while i < raw.len() { + let b = raw[i]; + if b == ENCODED_SLASH_SENTINEL { + // Raw sentinel byte in input — already rejected by the C0 + // control-byte sweep above, but double-check here to avoid + // collisions in case the sweep is ever relaxed. + return Err(CanonicalizeError::NullOrControlByte); + } + if b == b'%' { + if i + 2 >= raw.len() { + return Err(CanonicalizeError::InvalidPercentEncoding); + } + let decoded = match (decode_hex(raw[i + 1]), decode_hex(raw[i + 2])) { + (Some(hi), Some(lo)) => (hi << 4) | lo, + _ => return Err(CanonicalizeError::InvalidPercentEncoding), + }; + if decoded == b'/' { + if !allow_encoded_slash { + return Err(CanonicalizeError::EncodedSlashNotAllowed); + } + out.push(ENCODED_SLASH_SENTINEL); + } else if decoded == 0 || decoded == 0x7F || (decoded < 0x20 && decoded != b'\t') { + return Err(CanonicalizeError::NullOrControlByte); + } else if decoded == b'\n' || decoded == b'\r' || decoded == b'\t' { + // %-encoded CR/LF/TAB are still control bytes; reject. + return Err(CanonicalizeError::NullOrControlByte); + } else { + out.push(decoded); + } + i += 3; + } else { + out.push(b); + i += 1; + } + } + Ok(out) +} + +fn split_path_segments(decoded: &[u8]) -> Vec<&[u8]> { + // decoded is guaranteed to start with `/`. Skip the leading `/` and + // split on subsequent `/` bytes. The sentinel byte for encoded slashes + // never matches, so it stays inside its segment. + decoded[1..].split(|&b| b == b'/').collect() +} + +fn resolve_dot_segments(segments: Vec<&[u8]>) -> Result>, CanonicalizeError> { + let mut stack: Vec> = Vec::with_capacity(segments.len()); + let last = segments.len().saturating_sub(1); + for (idx, seg) in segments.into_iter().enumerate() { + if seg == b".." { + if stack.pop().is_none() { + return Err(CanonicalizeError::TraversalAboveRoot); + } + if idx == last { + // A trailing `..` leaves a "directory" (trailing slash). + stack.push(Vec::new()); + } + continue; + } + if seg == b"." { + if idx == last { + stack.push(Vec::new()); + } + continue; + } + if seg.is_empty() && idx != last { + // Collapse repeated slashes except at the very end, where an + // empty trailing segment encodes a trailing `/`. + continue; + } + stack.push(seg.to_vec()); + } + Ok(stack) +} + +fn build_canonical_path( + segments: &[Vec], + _trailing_slash_hint: bool, + opts: CanonicalizeOptions, +) -> String { + let mut out = String::from("/"); + for (idx, seg) in segments.iter().enumerate() { + if idx > 0 { + out.push('/'); + } + let trimmed: &[u8] = if opts.strip_path_parameters { + match seg.iter().position(|&b| b == b';') { + Some(pos) => &seg[..pos], + None => seg, + } + } else { + seg + }; + for &b in trimmed { + if b == ENCODED_SLASH_SENTINEL { + out.push_str("%2F"); + } else if is_pchar_unreserved(b) { + out.push(b as char); + } else { + out.push('%'); + out.push(upper_hex_nibble(b >> 4)); + out.push(upper_hex_nibble(b & 0x0F)); + } + } + } + out +} + +fn is_pchar_unreserved(b: u8) -> bool { + // RFC 3986 pchar without the percent-encoded slot — i.e. bytes we emit + // literally. Unreserved plus RFC 3986 sub-delims plus `:` and `@`. + b.is_ascii_alphanumeric() + || matches!( + b, + b'-' | b'.' + | b'_' + | b'~' + | b'!' + | b'$' + | b'&' + | b'\'' + | b'(' + | b')' + | b'*' + | b'+' + | b',' + | b';' + | b'=' + | b':' + | b'@' + ) +} + +fn decode_hex(b: u8) -> Option { + match b { + b'0'..=b'9' => Some(b - b'0'), + b'a'..=b'f' => Some(b - b'a' + 10), + b'A'..=b'F' => Some(b - b'A' + 10), + _ => None, + } +} + +fn upper_hex_nibble(n: u8) -> char { + match n { + 0..=9 => (b'0' + n) as char, + 10..=15 => (b'A' + (n - 10)) as char, + _ => unreachable!("nibble out of range"), + } +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +#[cfg(test)] +mod tests { + use super::*; + + fn canon(input: &str) -> Result { + let opts = CanonicalizeOptions::default(); + canonicalize_request_target(input, &opts).map(|(p, _)| p.path) + } + + fn canon_with(input: &str, opts: CanonicalizeOptions) -> Result { + canonicalize_request_target(input, &opts).map(|(p, _)| p.path) + } + + #[test] + fn literal_dot_segments_resolve() { + assert_eq!(canon("/a/./b").unwrap(), "/a/b"); + assert_eq!(canon("/a/b/.").unwrap(), "/a/b/"); + assert_eq!(canon("/a/../b").unwrap(), "/b"); + assert_eq!(canon("/a/b/..").unwrap(), "/a/"); + } + + #[test] + fn percent_encoded_dot_segments_resolve_the_same_way() { + assert_eq!(canon("/public/%2e%2e/secret").unwrap(), "/secret"); + assert_eq!(canon("/public/%2E%2E/secret").unwrap(), "/secret"); + assert_eq!(canon("/public/%2e/secret").unwrap(), "/public/secret"); + } + + #[test] + fn traversal_above_root_is_rejected() { + assert_eq!(canon("/.."), Err(CanonicalizeError::TraversalAboveRoot)); + assert_eq!( + canon("/a/../.."), + Err(CanonicalizeError::TraversalAboveRoot) + ); + assert_eq!( + canon("/a/%2e%2e/%2e%2e"), + Err(CanonicalizeError::TraversalAboveRoot) + ); + } + + #[test] + fn doubled_slashes_collapse() { + assert_eq!(canon("//").unwrap(), "/"); + assert_eq!(canon("//public//../secret").unwrap(), "/secret"); + assert_eq!(canon("/public//secret").unwrap(), "/public/secret"); + } + + #[test] + fn encoded_slash_rejected_by_default() { + assert_eq!( + canon("/a/%2f/b"), + Err(CanonicalizeError::EncodedSlashNotAllowed) + ); + assert_eq!( + canon("/public/..%2fsecret"), + Err(CanonicalizeError::EncodedSlashNotAllowed) + ); + } + + #[test] + fn encoded_slash_preserved_when_opted_in() { + let opts = CanonicalizeOptions { + allow_encoded_slash: true, + ..CanonicalizeOptions::default() + }; + assert_eq!(canon_with("/a/%2f/b", opts).unwrap(), "/a/%2F/b"); + assert_eq!(canon_with("/a/%2F/b", opts).unwrap(), "/a/%2F/b"); + } + + #[test] + fn null_and_control_bytes_rejected() { + assert_eq!(canon("/a%00b"), Err(CanonicalizeError::NullOrControlByte)); + assert_eq!(canon("/a%0Ab"), Err(CanonicalizeError::NullOrControlByte)); + assert_eq!(canon("/a%0Db"), Err(CanonicalizeError::NullOrControlByte)); + assert_eq!(canon("/a%7Fb"), Err(CanonicalizeError::NullOrControlByte)); + // Raw CR/LF/TAB in input should also fail. Build strings via + // byte-level concatenation since the literals in the source are + // otherwise flagged as control bytes in CI. + let mut raw = String::from("/a"); + raw.push('\n'); + raw.push('b'); + assert_eq!(canon(&raw), Err(CanonicalizeError::NullOrControlByte)); + } + + #[test] + fn fragment_rejected() { + assert_eq!( + canon("/a#b"), + Err(CanonicalizeError::FragmentInRequestTarget) + ); + } + + #[test] + fn absolute_form_strips_authority() { + assert_eq!(canon("http://host/a/../b").unwrap(), "/b"); + assert_eq!(canon("https://host").unwrap(), "/"); + assert_eq!(canon("http://host:443/foo").unwrap(), "/foo"); + } + + #[test] + fn legitimate_percent_encoded_bytes_round_trip() { + assert_eq!( + canon("/files/hello%20world.txt").unwrap(), + "/files/hello%20world.txt" + ); + assert_eq!(canon("/search/a%3Fb").unwrap(), "/search/a%3Fb"); + assert_eq!(canon("/users/%40alice").unwrap(), "/users/@alice"); + } + + #[test] + fn path_parameters_stripped_by_default() { + assert_eq!(canon("/a;jsessionid=xyz/b").unwrap(), "/a/b"); + assert_eq!(canon("/public;x=1/../secret").unwrap(), "/secret"); + } + + #[test] + fn path_parameters_preserved_when_disabled() { + let opts = CanonicalizeOptions { + strip_path_parameters: false, + ..CanonicalizeOptions::default() + }; + assert_eq!( + canon_with("/a;jsessionid=xyz/b", opts).unwrap(), + "/a;jsessionid=xyz/b" + ); + } + + #[test] + fn non_ascii_raw_byte_rejected() { + let mut raw = String::from("/a"); + raw.push('é'); + assert_eq!(canon(&raw), Err(CanonicalizeError::NonAscii)); + } + + #[test] + fn percent_encoded_non_ascii_bytes_round_trip() { + // `é` in UTF-8 is 0xC3 0xA9. The proxy treats the path as opaque + // bytes; percent-encoded high bytes pass through unchanged. + assert_eq!(canon("/users/caf%C3%A9").unwrap(), "/users/caf%C3%A9"); + } + + #[test] + fn empty_and_root_equivalent() { + assert_eq!(canon("").unwrap(), "/"); + assert_eq!(canon("/").unwrap(), "/"); + } + + #[test] + fn path_too_long_rejected() { + let long = format!("/{}", "a".repeat(MAX_PATH_LEN)); + assert_eq!(canon(&long), Err(CanonicalizeError::PathTooLong)); + } + + #[test] + fn mixed_case_percent_normalizes_to_uppercase() { + // Request comes in with lowercase %c3 — after canonicalization we + // emit %C3 so policy authors don't need to enumerate both cases. + assert_eq!(canon("/a/caf%c3%a9").unwrap(), "/a/caf%C3%A9"); + } + + #[test] + fn rewritten_flag_reflects_transformation() { + let (canon, _) = + canonicalize_request_target("/a", &CanonicalizeOptions::default()).unwrap(); + assert!(!canon.rewritten); + let (canon, _) = + canonicalize_request_target("/a/../b", &CanonicalizeOptions::default()).unwrap(); + assert!(canon.rewritten); + } + + #[test] + fn query_suffix_is_returned_separately() { + let (canon, query) = + canonicalize_request_target("/a?q=1&r=2", &CanonicalizeOptions::default()).unwrap(); + assert_eq!(canon.path, "/a"); + assert_eq!(query.as_deref(), Some("q=1&r=2")); + } + + // --------------------------------------------------------------------- + // Regression tests for the documented attack payloads. Every one of + // these used to bypass a `/public/**` allow rule because the proxy and + // the OPA policy never agreed with the upstream on what path was being + // dispatched. + // --------------------------------------------------------------------- + + #[test] + fn regression_public_slash_dotdot_secret() { + assert_eq!(canon("/public/../secret").unwrap(), "/secret"); + } + + #[test] + fn regression_public_slash_percent_dotdot_secret() { + assert_eq!(canon("/public/%2e%2e/secret").unwrap(), "/secret"); + assert_eq!(canon("/public/%2E%2E/secret").unwrap(), "/secret"); + } + + #[test] + fn regression_percent_encoded_slash_in_dotdot_rejected() { + assert_eq!( + canon("/public/%2E%2E%2Fsecret"), + Err(CanonicalizeError::EncodedSlashNotAllowed) + ); + } + + #[test] + fn regression_double_slash_prefix() { + assert_eq!(canon("//public/../secret").unwrap(), "/secret"); + } + + #[test] + fn regression_dot_slash_dotdot() { + assert_eq!(canon("/public/./../secret").unwrap(), "/secret"); + } +} diff --git a/crates/openshell-sandbox/src/l7/relay.rs b/crates/openshell-sandbox/src/l7/relay.rs index 110f777e91..d0599ea99f 100644 --- a/crates/openshell-sandbox/src/l7/relay.rs +++ b/crates/openshell-sandbox/src/l7/relay.rs @@ -9,15 +9,16 @@ use crate::l7::provider::{L7Provider, RelayOutcome}; use crate::l7::{EnforcementMode, L7EndpointConfig, L7Protocol, L7RequestInfo}; +use crate::opa::{PolicyGenerationGuard, TunnelPolicyEngine}; use crate::secrets::{self, SecretResolver}; use miette::{IntoDiagnostic, Result, miette}; use openshell_ocsf::{ ActionId, ActivityId, DispositionId, Endpoint, HttpActivityBuilder, HttpRequest, - NetworkActivityBuilder, SeverityId, Url as OcsfUrl, ocsf_emit, + NetworkActivityBuilder, SeverityId, StatusId, Url as OcsfUrl, ocsf_emit, }; -use std::sync::{Arc, Mutex}; +use std::sync::Arc; use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt}; -use tracing::{debug, info, warn}; +use tracing::{debug, warn}; /// Context for L7 request policy evaluation. pub struct L7EvalContext { @@ -37,6 +38,50 @@ pub struct L7EvalContext { pub(crate) secret_resolver: Option>, } +#[derive(Debug, Clone, Copy)] +enum ParseRejectionMode { + L7Endpoint, + Passthrough, +} + +fn parse_rejection_detail(error: &str, mode: ParseRejectionMode) -> String { + if error.contains("encoded '/' (%2F)") { + match mode { + ParseRejectionMode::L7Endpoint => format!( + "{error}; set allow_encoded_slash: true on this endpoint if the upstream requires encoded slashes" + ), + ParseRejectionMode::Passthrough => format!( + "{error}; passthrough credential relay uses strict path parsing, so configure this endpoint with protocol: rest and allow_encoded_slash: true for encoded-slash APIs, or use tls: skip if HTTP parsing is not needed" + ), + } + } else { + error.to_string() + } +} + +fn emit_parse_rejection(ctx: &L7EvalContext, detail: &str, engine_type: &str) { + let policy_name = if ctx.policy_name.is_empty() { + "-" + } else { + &ctx.policy_name + }; + let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Open) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) + .firewall_rule(policy_name, engine_type) + .message(format!( + "HTTP request rejected before policy evaluation for {}:{}", + ctx.host, ctx.port + )) + .status_detail(detail) + .build(); + ocsf_emit!(event); +} + /// Run protocol-aware L7 inspection on a tunnel. /// /// This replaces `copy_bidirectional` for L7-enabled endpoints. @@ -46,7 +91,7 @@ pub struct L7EvalContext { /// caller peeks on the raw `TcpStream` before calling this. pub async fn relay_with_inspection( config: &L7EndpointConfig, - engine: Mutex, + engine: TunnelPolicyEngine, client: &mut C, upstream: &mut U, ctx: &L7EvalContext, @@ -57,7 +102,11 @@ where { match config.protocol { L7Protocol::Rest => relay_rest(config, &engine, client, upstream, ctx).await, + L7Protocol::Graphql => relay_graphql(config, &engine, client, upstream, ctx).await, L7Protocol::Sql => { + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } // SQL provider is Phase 3 — fall through to passthrough with warning { let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) @@ -76,13 +125,254 @@ where } } +/// Run HTTP L7 inspection with per-request protocol selection. +/// +/// This is used when multiple L7 endpoints share a host:port, for example a +/// REST API under `/repos/**` and a GraphQL API under `/graphql`. +pub async fn relay_with_route_selection( + configs: &[L7EndpointConfig], + engine: TunnelPolicyEngine, + client: &mut C, + upstream: &mut U, + ctx: &L7EvalContext, +) -> Result<()> +where + C: AsyncRead + AsyncWrite + Unpin + Send, + U: AsyncRead + AsyncWrite + Unpin + Send, +{ + let provider = + crate::l7::rest::RestProvider::with_options(crate::l7::path::CanonicalizeOptions { + allow_encoded_slash: configs.iter().any(|config| config.allow_encoded_slash), + ..Default::default() + }); + + loop { + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + + let mut req = match provider.parse_request(client).await { + Ok(Some(req)) => req, + Ok(None) => return Ok(()), + Err(e) => { + if is_benign_connection_error(&e) { + debug!( + host = %ctx.host, + port = ctx.port, + error = %e, + "L7 route-selected connection closed" + ); + } else { + let detail = + parse_rejection_detail(&e.to_string(), ParseRejectionMode::L7Endpoint); + emit_parse_rejection(ctx, &detail, "l7"); + } + return Ok(()); + } + }; + + let Some(config) = select_l7_config_for_path(configs, &req.target) else { + crate::l7::rest::RestProvider::default() + .deny( + &req, + &ctx.policy_name, + "no L7 endpoint path matched request", + client, + ) + .await?; + return Ok(()); + }; + + let graphql_info = if config.protocol == L7Protocol::Graphql { + match crate::l7::graphql::inspect_graphql_request( + client, + &mut req, + config.graphql_max_body_bytes, + ) + .await + { + Ok(info) => Some(info), + Err(e) => { + if is_benign_connection_error(&e) { + debug!( + host = %ctx.host, + port = ctx.port, + error = %e, + "GraphQL L7 connection closed" + ); + } else { + let detail = + parse_rejection_detail(&e.to_string(), ParseRejectionMode::L7Endpoint); + emit_parse_rejection(ctx, &detail, "l7-graphql"); + } + return Ok(()); + } + } + } else { + None + }; + + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + + let (eval_target, redacted_target) = if let Some(ref resolver) = ctx.secret_resolver { + match secrets::rewrite_target_for_eval(&req.target, resolver) { + Ok(result) => (result.resolved, result.redacted), + Err(e) => { + warn!( + host = %ctx.host, + port = ctx.port, + error = %e, + "credential resolution failed in request target, rejecting" + ); + let response = b"HTTP/1.1 500 Internal Server Error\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"; + client.write_all(response).await.into_diagnostic()?; + client.flush().await.into_diagnostic()?; + return Ok(()); + } + } + } else { + (req.target.clone(), req.target.clone()) + }; + + let request_info = L7RequestInfo { + action: req.action.clone(), + target: redacted_target.clone(), + query_params: req.query_params.clone(), + graphql: graphql_info.clone(), + }; + + let parse_error_reason = graphql_info + .as_ref() + .and_then(|info| info.error.as_deref()) + .map(|error| format!("GraphQL request rejected: {error}")); + let force_deny = parse_error_reason.is_some(); + let (allowed, reason) = if let Some(reason) = parse_error_reason { + (false, reason) + } else { + evaluate_l7_request(&engine, ctx, &request_info)? + }; + + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + + let decision_str = match (allowed, config.enforcement) { + (_, _) if force_deny => "deny", + (true, _) => "allow", + (false, EnforcementMode::Audit) => "audit", + (false, EnforcementMode::Enforce) => "deny", + }; + let engine_type = if config.protocol == L7Protocol::Graphql { + "l7-graphql" + } else { + "l7" + }; + emit_l7_request_log( + ctx, + &request_info, + &redacted_target, + decision_str, + engine_type, + &reason, + graphql_info.as_ref(), + ); + + let _ = &eval_target; + + if allowed || (config.enforcement == EnforcementMode::Audit && !force_deny) { + let outcome = crate::l7::rest::relay_http_request_with_resolver_guarded( + &req, + client, + upstream, + ctx.secret_resolver.as_deref(), + Some(engine.generation_guard()), + ) + .await?; + match outcome { + RelayOutcome::Reusable => {} + RelayOutcome::Consumed => return Ok(()), + RelayOutcome::Upgraded { overflow } => { + return handle_upgrade(client, upstream, overflow, &ctx.host, ctx.port).await; + } + } + } else { + crate::l7::rest::RestProvider::default() + .deny_with_redacted_target( + &req, + &ctx.policy_name, + &reason, + client, + Some(&redacted_target), + ) + .await?; + return Ok(()); + } + } +} + +fn select_l7_config_for_path<'a>( + configs: &'a [L7EndpointConfig], + path: &str, +) -> Option<&'a L7EndpointConfig> { + configs + .iter() + .filter(|config| config.matches_path(path)) + .max_by_key(|config| config.path_specificity()) +} + +fn emit_l7_request_log( + ctx: &L7EvalContext, + request_info: &L7RequestInfo, + redacted_target: &str, + decision_str: &str, + engine_type: &str, + reason: &str, + graphql_info: Option<&crate::l7::graphql::GraphqlRequestInfo>, +) { + let (action_id, disposition_id, severity) = match decision_str { + "deny" => (ActionId::Denied, DispositionId::Blocked, SeverityId::Medium), + "allow" | "audit" => ( + ActionId::Allowed, + DispositionId::Allowed, + SeverityId::Informational, + ), + _ => ( + ActionId::Other, + DispositionId::Other, + SeverityId::Informational, + ), + }; + let summary = graphql_info + .map(|info| format!(" {}", graphql_log_summary(info))) + .unwrap_or_default(); + let event = HttpActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Other) + .action(action_id) + .disposition(disposition_id) + .severity(severity) + .http_request(HttpRequest::new( + &request_info.action, + OcsfUrl::new("http", &ctx.host, redacted_target, ctx.port), + )) + .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) + .firewall_rule(&ctx.policy_name, engine_type) + .message(format!( + "L7_REQUEST {decision_str} {} {}:{}{}{} reason={}", + request_info.action, ctx.host, ctx.port, redacted_target, summary, reason, + )) + .build(); + ocsf_emit!(event); +} + /// Handle an upgraded connection (101 Switching Protocols). /// /// Forwards any overflow bytes from the upgrade response to the client, then /// switches to raw bidirectional TCP copy for the upgraded protocol (WebSocket, /// HTTP/2, etc.). L7 policy enforcement does not apply after the upgrade — /// the initial HTTP request was already evaluated. -async fn handle_upgrade( +pub(crate) async fn handle_upgrade( client: &mut C, upstream: &mut U, overflow: Vec, @@ -119,7 +409,7 @@ where /// REST relay loop: parse request -> evaluate -> allow/deny -> relay response -> repeat. async fn relay_rest( config: &L7EndpointConfig, - engine: &Mutex, + engine: &TunnelPolicyEngine, client: &mut C, upstream: &mut U, ctx: &L7EvalContext, @@ -128,9 +418,21 @@ where C: AsyncRead + AsyncWrite + Unpin + Send, U: AsyncRead + AsyncWrite + Unpin + Send, { + // Build a provider carrying the per-endpoint canonicalization options so + // request parsing honors the endpoint's `allow_encoded_slash` setting + // (e.g. APIs like GitLab that embed `%2F` in path segments). + let provider = + crate::l7::rest::RestProvider::with_options(crate::l7::path::CanonicalizeOptions { + allow_encoded_slash: config.allow_encoded_slash, + ..Default::default() + }); loop { + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + // Parse one HTTP request from client - let req = match crate::l7::rest::RestProvider.parse_request(client).await { + let req = match provider.parse_request(client).await { Ok(Some(req)) => req, Ok(None) => return Ok(()), // Client closed connection Err(e) => { @@ -142,18 +444,18 @@ where "L7 connection closed" ); } else { - let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Low) - .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) - .message(format!("HTTP parse error in L7 relay: {e}")) - .build(); - ocsf_emit!(event); + let detail = + parse_rejection_detail(&e.to_string(), ParseRejectionMode::L7Endpoint); + emit_parse_rejection(ctx, &detail, "l7"); } return Ok(()); // Close connection on parse error } }; + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + // Rewrite credential placeholders in the request target BEFORE OPA // evaluation. OPA sees the redacted path; the resolved path goes only // to the upstream write. @@ -181,11 +483,16 @@ where action: req.action.clone(), target: redacted_target.clone(), query_params: req.query_params.clone(), + graphql: None, }; // Evaluate L7 policy via Rego (using redacted target) let (allowed, reason) = evaluate_l7_request(engine, ctx, &request_info)?; + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + // Check if this is an upgrade request for logging purposes. let header_end = req .raw_header @@ -210,13 +517,8 @@ where // Uses redacted_target (path only, no query params) to avoid logging secrets. { let (action_id, disposition_id, severity) = match decision_str { - "allow" => ( - ActionId::Allowed, - DispositionId::Allowed, - SeverityId::Informational, - ), "deny" => (ActionId::Denied, DispositionId::Blocked, SeverityId::Medium), - "audit" => ( + "allow" | "audit" => ( ActionId::Allowed, DispositionId::Allowed, SeverityId::Informational, @@ -251,11 +553,12 @@ where if allowed || config.enforcement == EnforcementMode::Audit { // Forward request to upstream and relay response - let outcome = crate::l7::rest::relay_http_request_with_resolver( + let outcome = crate::l7::rest::relay_http_request_with_resolver_guarded( &req, client, upstream, ctx.secret_resolver.as_deref(), + Some(engine.generation_guard()), ) .await?; match outcome { @@ -274,7 +577,212 @@ where } } else { // Enforce mode: deny with 403 and close connection (use redacted target) - crate::l7::rest::RestProvider + provider + .deny_with_redacted_target( + &req, + &ctx.policy_name, + &reason, + client, + Some(&redacted_target), + ) + .await?; + return Ok(()); + } + } +} + +fn close_if_stale(guard: &PolicyGenerationGuard, ctx: &L7EvalContext) -> bool { + if !guard.is_stale() { + return false; + } + + ocsf_emit!( + NetworkActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Open) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) + .firewall_rule(&ctx.policy_name, "l7") + .message(format!( + "L7 tunnel closed after policy reload [host:{} port:{} captured_generation:{} current_generation:{}]", + ctx.host, + ctx.port, + guard.captured_generation(), + guard.current_generation(), + )) + .build() + ); + true +} + +async fn relay_graphql( + config: &L7EndpointConfig, + engine: &TunnelPolicyEngine, + client: &mut C, + upstream: &mut U, + ctx: &L7EvalContext, +) -> Result<()> +where + C: AsyncRead + AsyncWrite + Unpin + Send, + U: AsyncRead + AsyncWrite + Unpin + Send, +{ + loop { + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + + let parsed = match crate::l7::graphql::parse_graphql_http_request( + client, + config.graphql_max_body_bytes, + crate::l7::path::CanonicalizeOptions { + allow_encoded_slash: config.allow_encoded_slash, + ..Default::default() + }, + ) + .await + { + Ok(Some(parsed)) => parsed, + Ok(None) => return Ok(()), + Err(e) => { + if is_benign_connection_error(&e) { + debug!( + host = %ctx.host, + port = ctx.port, + error = %e, + "GraphQL L7 connection closed" + ); + } else { + let detail = + parse_rejection_detail(&e.to_string(), ParseRejectionMode::L7Endpoint); + emit_parse_rejection(ctx, &detail, "l7-graphql"); + } + return Ok(()); + } + }; + + let req = parsed.request; + let graphql_info = parsed.info; + + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + + let (eval_target, redacted_target) = if let Some(ref resolver) = ctx.secret_resolver { + match secrets::rewrite_target_for_eval(&req.target, resolver) { + Ok(result) => (result.resolved, result.redacted), + Err(e) => { + warn!( + host = %ctx.host, + port = ctx.port, + error = %e, + "credential resolution failed in GraphQL request target, rejecting" + ); + let response = b"HTTP/1.1 500 Internal Server Error\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"; + client.write_all(response).await.into_diagnostic()?; + client.flush().await.into_diagnostic()?; + return Ok(()); + } + } + } else { + (req.target.clone(), req.target.clone()) + }; + + let request_info = L7RequestInfo { + action: req.action.clone(), + target: redacted_target.clone(), + query_params: req.query_params.clone(), + graphql: Some(graphql_info.clone()), + }; + + // Malformed or ambiguous GraphQL requests, such as duplicated GET + // control parameters, are rejected before policy evaluation. This + // keeps parser-differential cases fail-closed even if the endpoint is + // otherwise in audit mode. + let parse_error_reason = graphql_info + .error + .as_deref() + .map(|error| format!("GraphQL request rejected: {error}")); + let force_deny = parse_error_reason.is_some(); + let (allowed, reason) = if let Some(reason) = parse_error_reason { + (false, reason) + } else { + evaluate_l7_request(engine, ctx, &request_info)? + }; + + if close_if_stale(engine.generation_guard(), ctx) { + return Ok(()); + } + + let decision_str = match (allowed, config.enforcement) { + (_, _) if force_deny => "deny", + (true, _) => "allow", + (false, EnforcementMode::Audit) => "audit", + (false, EnforcementMode::Enforce) => "deny", + }; + + { + let (action_id, disposition_id, severity) = match decision_str { + "deny" => (ActionId::Denied, DispositionId::Blocked, SeverityId::Medium), + "allow" | "audit" => ( + ActionId::Allowed, + DispositionId::Allowed, + SeverityId::Informational, + ), + _ => ( + ActionId::Other, + DispositionId::Other, + SeverityId::Informational, + ), + }; + let gql_summary = graphql_log_summary(&graphql_info); + let event = HttpActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Other) + .action(action_id) + .disposition(disposition_id) + .severity(severity) + .http_request(HttpRequest::new( + &request_info.action, + OcsfUrl::new("http", &ctx.host, &redacted_target, ctx.port), + )) + .dst_endpoint(Endpoint::from_domain(&ctx.host, ctx.port)) + .firewall_rule(&ctx.policy_name, "l7-graphql") + .message(format!( + "GRAPHQL_L7_REQUEST {decision_str} {} {}:{}{} {gql_summary} reason={}", + request_info.action, ctx.host, ctx.port, redacted_target, reason, + )) + .build(); + ocsf_emit!(event); + } + + let _ = &eval_target; + + if allowed || (config.enforcement == EnforcementMode::Audit && !force_deny) { + let outcome = crate::l7::rest::relay_http_request_with_resolver_guarded( + &req, + client, + upstream, + ctx.secret_resolver.as_deref(), + Some(engine.generation_guard()), + ) + .await?; + match outcome { + RelayOutcome::Reusable => {} + RelayOutcome::Consumed => { + debug!( + host = %ctx.host, + port = ctx.port, + "Upstream connection not reusable, closing GraphQL L7 relay" + ); + return Ok(()); + } + RelayOutcome::Upgraded { overflow } => { + return handle_upgrade(client, upstream, overflow, &ctx.host, ctx.port).await; + } + } + } else { + crate::l7::rest::RestProvider::default() .deny_with_redacted_target( &req, &ctx.policy_name, @@ -288,6 +796,34 @@ where } } +fn graphql_log_summary(info: &crate::l7::graphql::GraphqlRequestInfo) -> String { + if let Some(error) = &info.error { + return format!("graphql_error={error:?}"); + } + let ops: Vec = info + .operations + .iter() + .map(|op| { + let name = op.operation_name.as_deref().unwrap_or("-"); + let fields = if op.fields.is_empty() { + "-".to_string() + } else { + op.fields.join(",") + }; + let persisted = op + .persisted_query_hash + .as_deref() + .or(op.persisted_query_id.as_deref()) + .unwrap_or("-"); + format!( + "type={} name={} fields={} persisted={}", + op.operation_type, name, fields, persisted + ) + }) + .collect(); + format!("graphql_ops={}", ops.join(";")) +} + /// Check if a miette error represents a benign connection close. /// /// TLS handshake EOF, missing `close_notify`, connection resets, and broken @@ -310,10 +846,18 @@ fn is_benign_connection_error(err: &miette::Report) -> bool { /// /// Returns `(allowed, deny_reason)`. pub fn evaluate_l7_request( - engine: &Mutex, + engine: &TunnelPolicyEngine, ctx: &L7EvalContext, request: &L7RequestInfo, ) -> Result<(bool, String)> { + if engine.is_stale() { + return Err(miette!( + "L7 tunnel policy generation is stale [captured_generation:{} current_generation:{}]", + engine.captured_generation(), + engine.current_generation(), + )); + } + let input_json = serde_json::json!({ "network": { "host": ctx.host, @@ -328,10 +872,12 @@ pub fn evaluate_l7_request( "method": request.action, "path": request.target, "query_params": request.query_params.clone(), + "graphql": request.graphql.clone(), } }); let mut engine = engine + .engine() .lock() .map_err(|_| miette!("OPA engine lock poisoned"))?; @@ -369,16 +915,25 @@ pub async fn relay_passthrough_with_credentials( client: &mut C, upstream: &mut U, ctx: &L7EvalContext, + generation_guard: &PolicyGenerationGuard, ) -> Result<()> where C: AsyncRead + AsyncWrite + Unpin + Send, U: AsyncRead + AsyncWrite + Unpin + Send, { - let provider = crate::l7::rest::RestProvider; + // Passthrough path: no L7 policy is enforced here, so use default + // (strict) canonicalization options. Calls to GitLab-style APIs that + // need `%2F` must be configured as L7 endpoints so the per-endpoint + // `allow_encoded_slash` opt-in applies. + let provider = crate::l7::rest::RestProvider::default(); let mut request_count: u64 = 0; let resolver = ctx.secret_resolver.as_deref(); loop { + if close_if_stale(generation_guard, ctx) { + return Ok(()); + } + // Read next request from client. let req = match provider.parse_request(client).await { Ok(Some(req)) => req, @@ -387,10 +942,17 @@ where if is_benign_connection_error(&e) { break; } - return Err(e); + let detail = + parse_rejection_detail(&e.to_string(), ParseRejectionMode::Passthrough); + emit_parse_rejection(ctx, &detail, "http-parser"); + return Ok(()); } }; + if close_if_stale(generation_guard, ctx) { + return Ok(()); + } + request_count += 1; // Resolve and redact the target for logging. @@ -439,9 +1001,14 @@ where // Forward request with credential rewriting and relay the response. // relay_http_request_with_resolver handles both directions: it sends // the request upstream and reads the response back to the client. - let outcome = - crate::l7::rest::relay_http_request_with_resolver(&req, client, upstream, resolver) - .await?; + let outcome = crate::l7::rest::relay_http_request_with_resolver_guarded( + &req, + client, + upstream, + resolver, + Some(generation_guard), + ) + .await?; match outcome { RelayOutcome::Reusable => {} // continue loop @@ -461,3 +1028,263 @@ where Ok(()) } + +#[cfg(test)] +mod tests { + use super::*; + use crate::opa::{NetworkInput, OpaEngine}; + use std::path::PathBuf; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + const TEST_POLICY: &str = include_str!("../../data/sandbox-policy.rego"); + + #[test] + fn parse_rejection_detail_adds_l7_hint_for_encoded_slash() { + let detail = parse_rejection_detail( + "HTTP request-target rejected: request-target contains an encoded '/' (%2F) which is not allowed on this endpoint", + ParseRejectionMode::L7Endpoint, + ); + + assert!(detail.contains("allow_encoded_slash: true")); + assert!(detail.contains("upstream requires encoded slashes")); + } + + #[test] + fn parse_rejection_detail_adds_passthrough_hint_for_encoded_slash() { + let detail = parse_rejection_detail( + "HTTP request-target rejected: request-target contains an encoded '/' (%2F) which is not allowed on this endpoint", + ParseRejectionMode::Passthrough, + ); + + assert!(detail.contains("protocol: rest")); + assert!(detail.contains("allow_encoded_slash: true")); + assert!(detail.contains("tls: skip")); + } + + #[test] + fn parse_rejection_detail_preserves_other_errors() { + let error = "HTTP headers contain invalid UTF-8"; + + assert_eq!( + parse_rejection_detail(error, ParseRejectionMode::L7Endpoint), + error + ); + } + + #[tokio::test] + async fn l7_relay_closes_keep_alive_tunnel_after_policy_generation_change() { + let initial_data = r#" +network_policies: + rest_api: + name: rest_api + endpoints: + - host: api.example.test + port: 8080 + protocol: rest + enforcement: enforce + rules: + - allow: + method: POST + path: "/write" + binaries: + - { path: /usr/bin/curl } +"#; + let reloaded_data = r#" +network_policies: + rest_api: + name: rest_api + endpoints: + - host: api.example.test + port: 8080 + protocol: rest + enforcement: enforce + rules: + - allow: + method: GET + path: "/write" + binaries: + - { path: /usr/bin/curl } +"#; + let engine = OpaEngine::from_strings(TEST_POLICY, initial_data).unwrap(); + let input = NetworkInput { + host: "api.example.test".into(), + port: 8080, + binary_path: PathBuf::from("/usr/bin/curl"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + let (endpoint_config, generation) = engine + .query_endpoint_config_with_generation(&input) + .unwrap(); + let config = crate::l7::parse_l7_config(&endpoint_config.unwrap()).unwrap(); + let tunnel_engine = engine.clone_engine_for_tunnel(generation).unwrap(); + let ctx = L7EvalContext { + host: "api.example.test".into(), + port: 8080, + policy_name: "rest_api".into(), + binary_path: "/usr/bin/curl".into(), + ancestors: vec![], + cmdline_paths: vec![], + secret_resolver: None, + }; + + let (mut app, mut relay_client) = tokio::io::duplex(8192); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); + let relay = tokio::spawn(async move { + relay_with_inspection( + &config, + tunnel_engine, + &mut relay_client, + &mut relay_upstream, + &ctx, + ) + .await + }); + + app.write_all( + b"POST /write HTTP/1.1\r\nHost: api.example.test\r\nContent-Length: 0\r\nConnection: keep-alive\r\n\r\n", + ) + .await + .unwrap(); + + let mut first_upstream = [0u8; 512]; + let n = tokio::time::timeout( + std::time::Duration::from_secs(1), + upstream.read(&mut first_upstream), + ) + .await + .expect("first request should reach upstream") + .unwrap(); + let first_upstream = String::from_utf8_lossy(&first_upstream[..n]); + assert!(first_upstream.starts_with("POST /write HTTP/1.1")); + + upstream + .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: keep-alive\r\n\r\nOK") + .await + .unwrap(); + + let mut first_response = [0u8; 512]; + let n = tokio::time::timeout( + std::time::Duration::from_secs(1), + app.read(&mut first_response), + ) + .await + .expect("first response should reach client") + .unwrap(); + let first_response = String::from_utf8_lossy(&first_response[..n]); + assert!(first_response.contains("200 OK")); + + engine.reload(TEST_POLICY, reloaded_data).unwrap(); + app.write_all( + b"POST /write HTTP/1.1\r\nHost: api.example.test\r\nContent-Length: 0\r\nConnection: keep-alive\r\n\r\n", + ) + .await + .unwrap(); + + tokio::time::timeout(std::time::Duration::from_secs(1), relay) + .await + .expect("relay should close stale tunnel") + .unwrap() + .unwrap(); + + let mut second_upstream = [0u8; 128]; + let n = tokio::time::timeout( + std::time::Duration::from_secs(1), + upstream.read(&mut second_upstream), + ) + .await + .expect("upstream side should close") + .unwrap(); + assert_eq!(n, 0, "stale request must not be forwarded upstream"); + } + + #[tokio::test] + async fn passthrough_relay_closes_keep_alive_tunnel_after_policy_generation_change() { + let policy_data = "network_policies: {}\n"; + let engine = OpaEngine::from_strings(TEST_POLICY, policy_data).unwrap(); + let generation_guard = engine + .generation_guard(engine.current_generation()) + .unwrap(); + let ctx = L7EvalContext { + host: "api.example.test".into(), + port: 8080, + policy_name: "rest_api".into(), + binary_path: "/usr/bin/curl".into(), + ancestors: vec![], + cmdline_paths: vec![], + secret_resolver: None, + }; + + let (mut app, mut relay_client) = tokio::io::duplex(8192); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); + let relay = tokio::spawn(async move { + relay_passthrough_with_credentials( + &mut relay_client, + &mut relay_upstream, + &ctx, + &generation_guard, + ) + .await + }); + + app.write_all( + b"GET /first HTTP/1.1\r\nHost: api.example.test\r\nConnection: keep-alive\r\n\r\n", + ) + .await + .unwrap(); + + let mut first_upstream = [0u8; 512]; + let n = tokio::time::timeout( + std::time::Duration::from_secs(1), + upstream.read(&mut first_upstream), + ) + .await + .expect("first passthrough request should reach upstream") + .unwrap(); + let first_upstream = String::from_utf8_lossy(&first_upstream[..n]); + assert!(first_upstream.starts_with("GET /first HTTP/1.1")); + + upstream + .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: keep-alive\r\n\r\nOK") + .await + .unwrap(); + + let mut first_response = [0u8; 512]; + let n = tokio::time::timeout( + std::time::Duration::from_secs(1), + app.read(&mut first_response), + ) + .await + .expect("first passthrough response should reach client") + .unwrap(); + let first_response = String::from_utf8_lossy(&first_response[..n]); + assert!(first_response.contains("200 OK")); + + engine.reload(TEST_POLICY, policy_data).unwrap(); + app.write_all( + b"GET /second HTTP/1.1\r\nHost: api.example.test\r\nConnection: keep-alive\r\n\r\n", + ) + .await + .unwrap(); + + tokio::time::timeout(std::time::Duration::from_secs(1), relay) + .await + .expect("passthrough relay should close stale tunnel") + .unwrap() + .unwrap(); + + let mut second_upstream = [0u8; 128]; + let n = tokio::time::timeout( + std::time::Duration::from_secs(1), + upstream.read(&mut second_upstream), + ) + .await + .expect("upstream side should close") + .unwrap(); + assert_eq!( + n, 0, + "stale passthrough request must not be forwarded upstream" + ); + } +} diff --git a/crates/openshell-sandbox/src/l7/rest.rs b/crates/openshell-sandbox/src/l7/rest.rs index 6bbf7be4ee..19acdbf32e 100644 --- a/crates/openshell-sandbox/src/l7/rest.rs +++ b/crates/openshell-sandbox/src/l7/rest.rs @@ -8,11 +8,12 @@ //! and chunked transfer encoding for body framing. use crate::l7::provider::{BodyLength, L7Provider, L7Request, RelayOutcome}; +use crate::opa::PolicyGenerationGuard; use crate::secrets::rewrite_http_header_block; use miette::{IntoDiagnostic, Result, miette}; use std::collections::HashMap; use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; -use tracing::{debug, warn}; +use tracing::debug; const MAX_HEADER_BYTES: usize = 16384; // 16 KiB for HTTP headers const RELAY_BUF_SIZE: usize = 8192; @@ -22,14 +23,33 @@ const RELAY_BUF_SIZE: usize = 8192; const RELAY_EOF_IDLE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); /// HTTP/1.1 REST protocol provider. -pub struct RestProvider; +/// +/// Carries the path-canonicalization options derived from the endpoint +/// config so that different endpoints (e.g. one backed by GitLab that needs +/// `%2F` in paths and one backed by a strict API) can apply different +/// canonicalization strictness to the same `RestProvider` call surface. +#[derive(Debug, Clone, Default)] +pub struct RestProvider { + canonicalize_options: crate::l7::path::CanonicalizeOptions, +} + +impl RestProvider { + /// Construct a provider with explicit canonicalization options. Used by + /// `relay_rest` so endpoint config can opt in to looser behavior such + /// as `allow_encoded_slash`. + pub fn with_options(canonicalize_options: crate::l7::path::CanonicalizeOptions) -> Self { + Self { + canonicalize_options, + } + } +} impl L7Provider for RestProvider { async fn parse_request( &self, client: &mut C, ) -> Result> { - parse_http_request(client).await + parse_http_request(client, &self.canonicalize_options).await } async fn relay( @@ -78,7 +98,10 @@ impl RestProvider { /// forwarded upstream without L7 policy evaluation -- a request /// smuggling vulnerability. Byte-at-a-time overhead is negligible for /// the typical 200-800 byte headers on L7-inspected REST endpoints. -async fn parse_http_request(client: &mut C) -> Result> { +async fn parse_http_request( + client: &mut C, + canonicalize_options: &crate::l7::path::CanonicalizeOptions, +) -> Result> { let mut buf = Vec::with_capacity(4096); loop { @@ -149,17 +172,68 @@ async fn parse_http_request(client: &mut C) -> Result parse_query_params(q)?, + None => HashMap::new(), + }; + + if canonical.rewritten { + buf = rewrite_request_line_target( + &buf, + &method, + &canonical.path, + raw_query.as_deref(), + version, + )?; + } Ok(Some(L7Request { action: method, - target: path, + target: canonical.path, query_params, raw_header: buf, // exact header bytes up to and including \r\n\r\n body_length, })) } +/// Rebuild the request line in a raw HTTP header block with a canonicalized +/// target. Called when the canonical path differs from what the client sent, +/// so the upstream dispatches on the exact bytes the policy engine evaluated. +fn rewrite_request_line_target( + raw: &[u8], + method: &str, + canonical_path: &str, + raw_query: Option<&str>, + version: &str, +) -> Result> { + let eol = raw + .windows(2) + .position(|w| w == b"\r\n") + .ok_or_else(|| miette!("request line missing CRLF"))?; + let rest = &raw[eol..]; + let new_target = match raw_query { + Some(q) if !q.is_empty() => format!("{canonical_path}?{q}"), + _ => canonical_path.to_string(), + }; + let new_request_line = format!("{method} {new_target} {version}"); + let mut out = Vec::with_capacity(new_request_line.len() + rest.len()); + out.extend_from_slice(new_request_line.as_bytes()); + out.extend_from_slice(rest); + Ok(out) +} + pub(crate) fn parse_target_query(target: &str) -> Result<(String, HashMap>)> { match target.split_once('?') { Some((path, query)) => Ok((path.to_string(), parse_query_params(query)?)), @@ -167,7 +241,7 @@ pub(crate) fn parse_target_query(target: &str) -> Result<(String, HashMap Result>> { +pub(crate) fn parse_query_params(query: &str) -> Result>> { let mut params: HashMap> = HashMap::new(); if query.is_empty() { return Ok(params); @@ -261,6 +335,20 @@ pub(crate) async fn relay_http_request_with_resolver( upstream: &mut U, resolver: Option<&crate::secrets::SecretResolver>, ) -> Result +where + C: AsyncRead + AsyncWrite + Unpin, + U: AsyncRead + AsyncWrite + Unpin, +{ + relay_http_request_with_resolver_guarded(req, client, upstream, resolver, None).await +} + +pub(crate) async fn relay_http_request_with_resolver_guarded( + req: &L7Request, + client: &mut C, + upstream: &mut U, + resolver: Option<&crate::secrets::SecretResolver>, + generation_guard: Option<&PolicyGenerationGuard>, +) -> Result where C: AsyncRead + AsyncWrite + Unpin, U: AsyncRead + AsyncWrite + Unpin, @@ -274,6 +362,10 @@ where let rewrite_result = rewrite_http_header_block(&req.raw_header[..header_end], resolver) .map_err(|e| miette!("credential injection failed: {e}"))?; + if let Some(guard) = generation_guard { + guard.ensure_current()?; + } + upstream .write_all(&rewrite_result.rewritten) .await @@ -281,6 +373,9 @@ where let overflow = &req.raw_header[header_end..]; if !overflow.is_empty() { + if let Some(guard) = generation_guard { + guard.ensure_current()?; + } upstream.write_all(overflow).await.into_diagnostic()?; } let overflow_len = overflow.len() as u64; @@ -289,11 +384,17 @@ where BodyLength::ContentLength(len) => { let remaining = len.saturating_sub(overflow_len); if remaining > 0 { - relay_fixed(client, upstream, remaining).await?; + relay_fixed(client, upstream, remaining, generation_guard).await?; } } BodyLength::Chunked => { - relay_chunked(client, upstream, &req.raw_header[header_end..]).await?; + relay_chunked( + client, + upstream, + &req.raw_header[header_end..], + generation_guard, + ) + .await?; } BodyLength::None => {} } @@ -385,7 +486,7 @@ async fn send_deny_response( /// Per RFC 7230 Section 3.3.3, rejects requests containing both /// `Content-Length` and `Transfer-Encoding` headers to prevent request /// smuggling via CL/TE ambiguity. -fn parse_body_length(headers: &str) -> Result { +pub(crate) fn parse_body_length(headers: &str) -> Result { let mut has_te_chunked = false; let mut cl_value: Option = None; @@ -402,12 +503,12 @@ fn parse_body_length(headers: &str) -> Result { let len: u64 = val .parse() .map_err(|_| miette!("Request contains invalid Content-Length value"))?; - if let Some(prev) = cl_value { - if prev != len { - return Err(miette!( - "Request contains multiple Content-Length headers with differing values ({prev} vs {len})" - )); - } + if let Some(prev) = cl_value + && prev != len + { + return Err(miette!( + "Request contains multiple Content-Length headers with differing values ({prev} vs {len})" + )); } cl_value = Some(len); } @@ -429,7 +530,12 @@ fn parse_body_length(headers: &str) -> Result { } /// Relay exactly `len` bytes from reader to writer. -async fn relay_fixed(reader: &mut R, writer: &mut W, len: u64) -> Result<()> +async fn relay_fixed( + reader: &mut R, + writer: &mut W, + len: u64, + generation_guard: Option<&PolicyGenerationGuard>, +) -> Result<()> where R: AsyncRead + Unpin, W: AsyncWrite + Unpin, @@ -446,6 +552,9 @@ where "Connection closed with {remaining} bytes remaining" )); } + if let Some(guard) = generation_guard { + guard.ensure_current()?; + } writer.write_all(&buf[..n]).await.into_diagnostic()?; remaining -= n as u64; } @@ -461,7 +570,12 @@ where /// `already_forwarded` are overflow bytes that were already written to the /// writer during header parsing. They are seeded into the parser buffer so /// termination can still be detected when boundaries span reads. -async fn relay_chunked(reader: &mut R, writer: &mut W, already_forwarded: &[u8]) -> Result<()> +async fn relay_chunked( + reader: &mut R, + writer: &mut W, + already_forwarded: &[u8], + generation_guard: Option<&PolicyGenerationGuard>, +) -> Result<()> where R: AsyncRead + Unpin, W: AsyncWrite + Unpin, @@ -485,6 +599,9 @@ where if n == 0 { return Err(miette!("Chunked body ended before chunk-size line")); } + if let Some(guard) = generation_guard { + guard.ensure_current()?; + } writer.write_all(&read_buf[..n]).await.into_diagnostic()?; parse_buf.extend_from_slice(&read_buf[..n]); }; @@ -514,6 +631,9 @@ where if n == 0 { return Err(miette!("Chunked body ended before trailer terminator")); } + if let Some(guard) = generation_guard { + guard.ensure_current()?; + } writer.write_all(&read_buf[..n]).await.into_diagnostic()?; parse_buf.extend_from_slice(&read_buf[..n]); }; @@ -547,6 +667,9 @@ where if n == 0 { return Err(miette!("Chunked body ended mid-chunk")); } + if let Some(guard) = generation_guard { + guard.ensure_current()?; + } writer.write_all(&read_buf[..n]).await.into_diagnostic()?; parse_buf.extend_from_slice(&read_buf[..n]); } @@ -572,25 +695,6 @@ fn find_crlf(buf: &[u8], start: usize) -> Option { .map(|offset| start + offset) } -/// Read and relay a full HTTP response (headers + body) from upstream to client. -/// -/// Returns a [`RelayOutcome`] indicating whether the connection is reusable, -/// consumed, or has been upgraded (101 Switching Protocols). -/// -/// Note: callers that receive `Upgraded` are responsible for switching to -/// raw bidirectional relay and forwarding the overflow bytes. -pub(crate) async fn relay_response_to_client( - upstream: &mut U, - client: &mut C, - request_method: &str, -) -> Result -where - U: AsyncRead + Unpin, - C: AsyncWrite + Unpin, -{ - relay_response(request_method, upstream, client).await -} - async fn relay_response( request_method: &str, upstream: &mut U, @@ -716,11 +820,11 @@ where BodyLength::ContentLength(len) => { let remaining = len.saturating_sub(overflow_len); if remaining > 0 { - relay_fixed(upstream, client, remaining).await?; + relay_fixed(upstream, client, remaining, None).await?; } } BodyLength::Chunked => { - relay_chunked(upstream, client, &buf[header_end..]).await?; + relay_chunked(upstream, client, &buf[header_end..], None).await?; } BodyLength::None => unreachable!(), } @@ -858,11 +962,21 @@ fn is_benign_close(err: &std::io::Error) -> bool { } #[cfg(test)] +#[allow( + clippy::iter_on_single_items, + clippy::manual_string_new, + clippy::collapsible_if, + clippy::cast_possible_truncation, + reason = "Test code: test fixtures and explicit value-shape assertions are idiomatic in tests." +)] mod tests { use super::*; + use crate::opa::OpaEngine; use crate::secrets::SecretResolver; use base64::Engine as _; + const TEST_POLICY: &str = include_str!("../../data/sandbox-policy.rego"); + #[test] fn parse_content_length() { let headers = "POST /api HTTP/1.1\r\nHost: example.com\r\nContent-Length: 42\r\n\r\n"; @@ -1064,7 +1178,11 @@ mod tests { .await .unwrap(); }); - let result = parse_http_request(&mut client).await; + let result = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await; assert!(result.is_err(), "Must reject headers with bare LF"); } @@ -1077,7 +1195,11 @@ mod tests { raw.extend_from_slice(b"GET /api HTTP/1.1\r\nHost: x\r\nX-Bad: \xc0\xaf\r\n\r\n"); writer.write_all(&raw).await.unwrap(); }); - let result = parse_http_request(&mut client).await; + let result = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await; assert!(result.is_err(), "Must reject headers with invalid UTF-8"); } @@ -1091,10 +1213,180 @@ mod tests { .await .unwrap(); }); - let result = parse_http_request(&mut client).await; + let result = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await; assert!(result.is_err(), "Must reject unsupported HTTP version"); } + #[tokio::test] + async fn parse_http_request_canonicalizes_target_and_rewrites_raw_header() { + let (mut client, mut writer) = tokio::io::duplex(4096); + tokio::spawn(async move { + writer + .write_all(b"GET /public/../secret HTTP/1.1\r\nHost: api.example.com\r\n\r\n") + .await + .unwrap(); + }); + let req = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await + .expect("request should parse") + .expect("request should exist"); + // Path fed to OPA evaluation is canonical. + assert_eq!(req.target, "/secret"); + // raw_header (forwarded byte-for-byte to upstream) is also canonical + // — this is the invariant the L7 canonicalization PR must uphold. + assert_eq!( + req.raw_header, b"GET /secret HTTP/1.1\r\nHost: api.example.com\r\n\r\n", + "outbound request line must carry the canonical path" + ); + } + + #[tokio::test] + async fn parse_http_request_canonicalization_preserves_query_string() { + let (mut client, mut writer) = tokio::io::duplex(4096); + tokio::spawn(async move { + writer + .write_all(b"GET /public/../v1/list?limit=10&sort=asc HTTP/1.1\r\nHost: h\r\n\r\n") + .await + .unwrap(); + }); + let req = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(req.target, "/v1/list"); + assert_eq!( + req.raw_header, b"GET /v1/list?limit=10&sort=asc HTTP/1.1\r\nHost: h\r\n\r\n", + "canonical rewrite must preserve the query string verbatim" + ); + } + + #[tokio::test] + async fn parse_http_request_leaves_canonical_input_byte_for_byte() { + // When the input is already canonical, the raw_header must pass + // through unchanged — otherwise legitimate traffic pays a rewrite + // cost on every request. + let (mut client, mut writer) = tokio::io::duplex(4096); + tokio::spawn(async move { + writer + .write_all(b"GET /api/v1/users HTTP/1.1\r\nHost: api.example.com\r\n\r\n") + .await + .unwrap(); + }); + let req = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(req.target, "/api/v1/users"); + assert_eq!( + req.raw_header, + b"GET /api/v1/users HTTP/1.1\r\nHost: api.example.com\r\n\r\n", + ); + } + + #[tokio::test] + async fn parse_http_request_rejects_traversal_above_root() { + let (mut client, mut writer) = tokio::io::duplex(4096); + tokio::spawn(async move { + writer + .write_all(b"GET /.. HTTP/1.1\r\nHost: h\r\n\r\n") + .await + .unwrap(); + }); + let result = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await; + assert!( + result.is_err(), + "a target that escapes the path root must be rejected at the parser" + ); + } + + #[tokio::test] + async fn parse_http_request_accepts_encoded_slash_when_endpoint_opts_in() { + // GitLab-style endpoints legitimately embed `%2F` in path segments + // (e.g. `/api/v4/projects/group%2Fproject`). Passing a provider + // constructed with `allow_encoded_slash: true` models the + // endpoint-config wiring that flows from `L7EndpointConfig`. + let (mut client, mut writer) = tokio::io::duplex(4096); + tokio::spawn(async move { + writer + .write_all(b"GET /api/v4/projects/group%2Fproject HTTP/1.1\r\nHost: g\r\n\r\n") + .await + .unwrap(); + }); + let options = crate::l7::path::CanonicalizeOptions { + allow_encoded_slash: true, + ..Default::default() + }; + let req = parse_http_request(&mut client, &options) + .await + .unwrap() + .unwrap(); + assert_eq!(req.target, "/api/v4/projects/group%2Fproject"); + } + + #[tokio::test] + async fn parse_http_request_rejects_encoded_slash_by_default() { + // Default strict options must reject `%2F` — this is the security + // posture for endpoints where an encoded slash would let an + // attacker disagree with the upstream on segment boundaries. + let (mut client, mut writer) = tokio::io::duplex(4096); + tokio::spawn(async move { + writer + .write_all(b"GET /api/v4/projects/group%2Fproject HTTP/1.1\r\nHost: g\r\n\r\n") + .await + .unwrap(); + }); + let result = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await; + assert!( + result.is_err(), + "default options must reject encoded slashes in the path" + ); + } + + #[tokio::test] + async fn parse_http_request_preserves_http_10_version_on_rewrite() { + let (mut client, mut writer) = tokio::io::duplex(4096); + tokio::spawn(async move { + writer + .write_all(b"GET /a/./b HTTP/1.0\r\nHost: h\r\n\r\n") + .await + .unwrap(); + }); + let req = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(req.target, "/a/b"); + assert!( + req.raw_header.starts_with(b"GET /a/b HTTP/1.0\r\n"), + "rewrite must preserve the original HTTP version, got: {:?}", + String::from_utf8_lossy(&req.raw_header) + ); + } + #[tokio::test] async fn parse_http_request_splits_path_and_query_params() { let (mut client, mut writer) = tokio::io::duplex(4096); @@ -1106,10 +1398,13 @@ mod tests { .await .unwrap(); }); - let req = parse_http_request(&mut client) - .await - .expect("request should parse") - .expect("request should exist"); + let req = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await + .expect("request should parse") + .expect("request should exist"); assert_eq!(req.target, "/download"); assert_eq!( req.query_params.get("slug").cloned(), @@ -1139,10 +1434,13 @@ mod tests { .unwrap(); }); - let first = parse_http_request(&mut client) - .await - .expect("first request should parse") - .expect("expected first request"); + let first = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await + .expect("first request should parse") + .expect("expected first request"); assert_eq!(first.action, "GET"); assert_eq!(first.target, "/allowed"); assert!(first.query_params.is_empty()); @@ -1151,10 +1449,13 @@ mod tests { "raw_header must contain only the first request's headers" ); - let second = parse_http_request(&mut client) - .await - .expect("second request should parse") - .expect("expected second request"); + let second = parse_http_request( + &mut client, + &crate::l7::path::CanonicalizeOptions::default(), + ) + .await + .expect("second request should parse") + .expect("expected second request"); assert_eq!(second.action, "POST"); assert_eq!(second.target, "/blocked"); assert!(second.query_params.is_empty()); @@ -1697,6 +1998,47 @@ mod tests { upstream_task.await.expect("upstream task should complete"); } + #[tokio::test] + async fn relay_request_guard_blocks_stale_generation_before_upstream_write() { + let policy_data = "network_policies: {}\n"; + let engine = OpaEngine::from_strings(TEST_POLICY, policy_data).unwrap(); + let guard = engine + .generation_guard(engine.current_generation()) + .unwrap(); + engine.reload(TEST_POLICY, policy_data).unwrap(); + + let req = L7Request { + action: "GET".to_string(), + target: "/api".to_string(), + query_params: HashMap::new(), + raw_header: b"GET /api HTTP/1.1\r\nHost: example.com\r\n\r\n".to_vec(), + body_length: BodyLength::None, + }; + let (mut proxy_to_upstream, mut upstream_side) = tokio::io::duplex(8192); + let (mut _app_side, mut proxy_to_client) = tokio::io::duplex(8192); + + let result = relay_http_request_with_resolver_guarded( + &req, + &mut proxy_to_client, + &mut proxy_to_upstream, + None, + Some(&guard), + ) + .await; + assert!( + result.is_err(), + "stale generation must stop relay before upstream write" + ); + + drop(proxy_to_upstream); + let mut forwarded = Vec::new(); + upstream_side.read_to_end(&mut forwarded).await.unwrap(); + assert!( + forwarded.is_empty(), + "stale request bytes must not reach upstream" + ); + } + #[test] fn client_requested_upgrade_detects_websocket_headers() { let headers = "GET /ws HTTP/1.1\r\nHost: example.com\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\r\n"; diff --git a/crates/openshell-sandbox/src/l7/tls.rs b/crates/openshell-sandbox/src/l7/tls.rs index 4ec0de03cc..70e198f420 100644 --- a/crates/openshell-sandbox/src/l7/tls.rs +++ b/crates/openshell-sandbox/src/l7/tls.rs @@ -197,11 +197,28 @@ pub async fn tls_connect_upstream( Ok(tls_stream) } -/// Build a rustls `ClientConfig` with Mozilla root CAs for upstream connections. -pub fn build_upstream_client_config() -> Arc { +/// Build a rustls `ClientConfig` with Mozilla + system root CAs for upstream connections. +/// +/// `system_ca_bundle` is the pre-read PEM contents of the system CA bundle +/// (from [`read_system_ca_bundle`]). Pass the same string to [`write_ca_files`] +/// to avoid reading the bundle from disk twice. +pub fn build_upstream_client_config(system_ca_bundle: &str) -> Arc { let mut root_store = rustls::RootCertStore::empty(); root_store.extend(webpki_roots::TLS_SERVER_ROOTS.iter().cloned()); + // System bundles typically overlap with webpki-roots (Mozilla roots); + // duplicates are harmless and ensure we also pick up any custom/corporate CAs. + let (added, ignored) = load_pem_certs_into_store(&mut root_store, system_ca_bundle); + if added > 0 { + tracing::debug!(added, "Loaded system CA certificates for upstream TLS"); + } + if ignored > 0 { + tracing::warn!( + ignored, + "Some system CA certificates could not be parsed and were ignored" + ); + } + let mut config = ClientConfig::builder() .with_root_certificates(root_store) .with_no_client_auth(); @@ -216,15 +233,23 @@ pub fn build_upstream_client_config() -> Arc { /// 1. Standalone CA cert PEM (for `NODE_EXTRA_CA_CERTS` which is additive) /// 2. Combined bundle: system CAs + sandbox CA (for `SSL_CERT_FILE` which replaces default) /// +/// `system_ca_bundle` is the pre-read PEM contents of the system CA bundle +/// (from [`read_system_ca_bundle`]). Pass the same string to +/// [`build_upstream_client_config`] to avoid reading the bundle from disk twice. +/// /// Returns `(ca_cert_path, combined_bundle_path)`. -pub fn write_ca_files(ca: &SandboxCa, output_dir: &Path) -> Result<(PathBuf, PathBuf)> { +pub fn write_ca_files( + ca: &SandboxCa, + output_dir: &Path, + system_ca_bundle: &str, +) -> Result<(PathBuf, PathBuf)> { std::fs::create_dir_all(output_dir).into_diagnostic()?; let ca_cert_path = output_dir.join("openshell-ca.pem"); std::fs::write(&ca_cert_path, ca.cert_pem()).into_diagnostic()?; - // Read system CA bundle and append our CA - let mut combined = read_system_ca_bundle(); + // Combine system CAs with our sandbox CA + let mut combined = system_ca_bundle.to_string(); if !combined.is_empty() && !combined.ends_with('\n') { combined.push('\n'); } @@ -236,8 +261,36 @@ pub fn write_ca_files(ca: &SandboxCa, output_dir: &Path) -> Result<(PathBuf, Pat Ok((ca_cert_path, combined_path)) } +/// Load PEM-encoded certificates from a string into a root certificate store. +/// +/// Returns `(added, ignored)` counts. Invalid or unparseable certificates +/// are silently ignored, matching the behavior of +/// `RootCertStore::add_parsable_certificates`. +fn load_pem_certs_into_store( + root_store: &mut rustls::RootCertStore, + pem_data: &str, +) -> (usize, usize) { + if pem_data.is_empty() { + return (0, 0); + } + let mut reader = BufReader::new(pem_data.as_bytes()); + // Collect all results so we can count PEM blocks that fail base64 + // decoding — rustls_pemfile::certs silently drops those, so without + // this they wouldn't be reflected in the `ignored` count. + let all_results: Vec<_> = rustls_pemfile::certs(&mut reader).collect(); + let pem_errors = all_results.iter().filter(|r| r.is_err()).count(); + let certs: Vec> = + all_results.into_iter().filter_map(Result::ok).collect(); + let (added, ignored) = root_store.add_parsable_certificates(certs); + (added, ignored + pem_errors) +} + /// Read the system CA bundle from well-known paths. -fn read_system_ca_bundle() -> String { +/// +/// Returns the PEM contents of the first non-empty bundle found, or an empty +/// string if none of the well-known paths exist. Call once and pass the result +/// to both [`write_ca_files`] and [`build_upstream_client_config`]. +pub fn read_system_ca_bundle() -> String { for path in SYSTEM_CA_PATHS { if let Ok(contents) = std::fs::read_to_string(path) && !contents.is_empty() @@ -260,10 +313,10 @@ pub fn parse_pem_certs(path: &Path) -> Result>> { } /// Peek the first bytes of a stream and determine if it looks like a TLS -/// ClientHello handshake. +/// `ClientHello` handshake. /// /// A TLS record starts with: -/// - byte 0: `0x16` (ContentType::Handshake) +/// - byte 0: `0x16` (`ContentType::Handshake`) /// - bytes 1-2: TLS version (0x0301 = TLS 1.0, 0x0302 = TLS 1.1, 0x0303 = TLS 1.2/1.3) /// /// Returns `true` if the peeked bytes match the TLS handshake pattern. @@ -373,7 +426,97 @@ mod tests { #[test] fn upstream_config_alpn() { let _ = rustls::crypto::ring::default_provider().install_default(); - let config = build_upstream_client_config(); + let config = build_upstream_client_config(""); assert_eq!(config.alpn_protocols, vec![b"http/1.1".to_vec()]); } + + /// Helper: generate a self-signed CA and return its PEM string. + fn generate_ca_pem() -> String { + SandboxCa::generate().unwrap().ca_cert_pem + } + + #[test] + fn load_pem_certs_single_ca() { + let pem = generate_ca_pem(); + let mut store = rustls::RootCertStore::empty(); + let (added, ignored) = load_pem_certs_into_store(&mut store, &pem); + assert_eq!(added, 1); + assert_eq!(ignored, 0); + } + + #[test] + fn load_pem_certs_multiple_cas() { + let bundle = format!( + "{}\n{}\n{}\n", + generate_ca_pem(), + generate_ca_pem(), + generate_ca_pem() + ); + let mut store = rustls::RootCertStore::empty(); + let (added, ignored) = load_pem_certs_into_store(&mut store, &bundle); + assert_eq!(added, 3); + assert_eq!(ignored, 0); + } + + #[test] + fn load_pem_certs_empty_string() { + let mut store = rustls::RootCertStore::empty(); + let (added, ignored) = load_pem_certs_into_store(&mut store, ""); + assert_eq!(added, 0); + assert_eq!(ignored, 0); + } + + #[test] + fn load_pem_certs_garbage_input() { + let mut store = rustls::RootCertStore::empty(); + let (added, ignored) = load_pem_certs_into_store(&mut store, "this is not PEM data at all"); + assert_eq!(added, 0); + assert_eq!(ignored, 0); + } + + #[test] + fn load_pem_certs_malformed_pem_block() { + let malformed = "-----BEGIN CERTIFICATE-----\nNOTBASE64!!!\n-----END CERTIFICATE-----\n"; + let mut store = rustls::RootCertStore::empty(); + let (added, ignored) = load_pem_certs_into_store(&mut store, malformed); + assert_eq!(added, 0); + assert_eq!(ignored, 1); + } + + #[test] + fn load_pem_certs_mixed_valid_and_invalid() { + let malformed = "-----BEGIN CERTIFICATE-----\nNOTBASE64!!!\n-----END CERTIFICATE-----\n"; + let bundle = format!( + "{}\n{}{}\n", + generate_ca_pem(), + malformed, + generate_ca_pem() + ); + let mut store = rustls::RootCertStore::empty(); + let (added, ignored) = load_pem_certs_into_store(&mut store, &bundle); + assert_eq!(added, 2); + assert_eq!(ignored, 1); + } + + #[test] + fn write_ca_files_includes_sandbox_ca() { + let ca = SandboxCa::generate().unwrap(); + let dir = tempfile::tempdir().unwrap(); + let (ca_path, bundle_path) = write_ca_files(&ca, dir.path(), "").unwrap(); + + // Standalone CA cert file should exist and be valid PEM + let ca_pem = std::fs::read_to_string(&ca_path).unwrap(); + assert!(ca_pem.starts_with("-----BEGIN CERTIFICATE-----")); + + // Combined bundle should contain at least the sandbox CA + let bundle_pem = std::fs::read_to_string(&bundle_path).unwrap(); + assert!(bundle_pem.contains(ca.cert_pem())); + + // Bundle should be parseable as PEM certificates + let mut reader = BufReader::new(bundle_pem.as_bytes()); + assert!( + rustls_pemfile::certs(&mut reader).any(|r| r.is_ok()), + "bundle should contain at least one cert", + ); + } } diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index c2956b1e09..19424bd2b0 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! OpenShell Sandbox library. +//! `OpenShell` Sandbox library. //! //! This crate provides process sandboxing and monitoring capabilities. @@ -21,16 +21,19 @@ pub mod proxy; mod sandbox; mod secrets; mod ssh; +mod supervisor_session; use miette::{IntoDiagnostic, Result}; #[cfg(target_os = "linux")] use std::collections::HashSet; +use std::future::Future; use std::net::SocketAddr; use std::sync::Arc; +use std::sync::LazyLock; +#[cfg(any(target_os = "linux", test))] +use std::sync::Mutex; use std::sync::OnceLock; use std::sync::atomic::{AtomicU32, Ordering}; -#[cfg(target_os = "linux")] -use std::sync::{LazyLock, Mutex}; use std::time::Duration; use tokio::time::timeout; use tracing::{debug, info, trace, warn}; @@ -66,16 +69,15 @@ static OCSF_CTX: OnceLock = OnceLock::new(); /// Fallback context used when `OCSF_CTX` has not been initialized (e.g. in /// unit tests that exercise individual functions without calling `run_sandbox`). -static OCSF_CTX_FALLBACK: std::sync::LazyLock = - std::sync::LazyLock::new(|| SandboxContext { - sandbox_id: String::new(), - sandbox_name: String::new(), - container_image: String::new(), - hostname: "test".to_string(), - product_version: openshell_core::VERSION.to_string(), - proxy_ip: std::net::IpAddr::from([127, 0, 0, 1]), - proxy_port: 3128, - }); +static OCSF_CTX_FALLBACK: LazyLock = LazyLock::new(|| SandboxContext { + sandbox_id: String::new(), + sandbox_name: String::new(), + container_image: String::new(), + hostname: "test".to_string(), + product_version: openshell_core::VERSION.to_string(), + proxy_ip: std::net::IpAddr::from([127, 0, 0, 1]), + proxy_port: 3128, +}); /// Return a reference to the process-wide [`SandboxContext`]. /// @@ -87,7 +89,8 @@ pub(crate) fn ocsf_ctx() -> &'static SandboxContext { use crate::identity::BinaryIdentityCache; use crate::l7::tls::{ - CertCache, ProxyTlsState, SandboxCa, build_upstream_client_config, write_ca_files, + CertCache, ProxyTlsState, SandboxCa, build_upstream_client_config, read_system_ca_bundle, + write_ca_files, }; use crate::opa::OpaEngine; use crate::policy::{NetworkMode, NetworkPolicy, ProxyPolicy, SandboxPolicy}; @@ -96,6 +99,7 @@ use crate::proxy::ProxyHandle; use crate::sandbox::linux::netns::NetworkNamespace; use crate::secrets::SecretResolver; pub use process::{ProcessHandle, ProcessStatus}; +pub use sandbox::apply_supervisor_startup_hardening; /// Default interval (seconds) for re-fetching the inference route bundle from /// the gateway in cluster mode. Override at runtime with the @@ -130,27 +134,27 @@ fn disable_inference_on_empty_routes(source: InferenceRouteSource) -> bool { } fn route_refresh_interval_secs() -> u64 { - match std::env::var("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS") { - Ok(value) => match value.parse::() { - Ok(interval) if interval > 0 => interval, - Ok(_) => { - warn!( - default_interval_secs = DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, - "Ignoring zero route refresh interval" - ); - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - } - Err(error) => { - warn!( - interval = %value, - error = %error, - default_interval_secs = DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, - "Ignoring invalid route refresh interval" - ); - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - } - }, - Err(_) => DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, + let Ok(value) = std::env::var("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS") else { + return DEFAULT_ROUTE_REFRESH_INTERVAL_SECS; + }; + match value.parse::() { + Ok(interval) if interval > 0 => interval, + Ok(_) => { + warn!( + default_interval_secs = DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, + "Ignoring zero route refresh interval" + ); + DEFAULT_ROUTE_REFRESH_INTERVAL_SECS + } + Err(error) => { + warn!( + interval = %value, + error = %error, + default_interval_secs = DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, + "Ignoring invalid route refresh interval" + ); + DEFAULT_ROUTE_REFRESH_INTERVAL_SECS + } } } @@ -207,7 +211,7 @@ pub async fn run_sandbox( openshell_endpoint: Option, policy_rules: Option, policy_data: Option, - ssh_listen_addr: Option, + ssh_socket_path: Option, ssh_handshake_secret: Option, ssh_handshake_skew_secs: u64, _health_check: bool, @@ -224,9 +228,10 @@ pub async fn run_sandbox( // Proxy IP/port use defaults here; they are only significant for network // events which happen after the netns is created. { - let hostname = std::fs::read_to_string("/etc/hostname") - .map(|s| s.trim().to_string()) - .unwrap_or_else(|_| "openshell-sandbox".to_string()); + let hostname = std::fs::read_to_string("/etc/hostname").map_or_else( + |_| "openshell-sandbox".to_string(), + |s| s.trim().to_string(), + ); if OCSF_CTX .set(SandboxContext { @@ -315,13 +320,14 @@ pub async fn run_sandbox( match SandboxCa::generate() { Ok(ca) => { let tls_dir = std::path::Path::new("/etc/openshell-tls"); - match write_ca_files(&ca, tls_dir) { + let system_ca_bundle = read_system_ca_bundle(); + match write_ca_files(&ca, tls_dir, &system_ca_bundle) { Ok(paths) => { // /etc/openshell-tls is subsumed by the /etc baseline // path injected by enrich_*_baseline_paths(), so no // explicit Landlock entry is needed here. - let upstream_config = build_upstream_client_config(); + let upstream_config = build_upstream_client_config(&system_ca_bundle); let cert_cache = CertCache::new(ca); let state = Arc::new(ProxyTlsState::new(cert_cache, upstream_config)); ocsf_emit!( @@ -414,6 +420,11 @@ pub async fn run_sandbox( #[allow(clippy::no_effect_underscore_binding)] let _netns: Option<()> = None; + // Install the supervisor seccomp prelude after privileged startup helpers + // (network namespace setup, iptables probes) complete, but before the SSH + // listener and workload process are exposed. + apply_supervisor_startup_hardening()?; + // Shared PID: set after process spawn so the proxy can look up // the entrypoint process's /proc/net/tcp for identity binding. let entrypoint_pid = Arc::new(AtomicU32::new(0)); @@ -482,15 +493,13 @@ pub async fn run_sandbox( // Reads /dev/kmsg for iptables LOG entries and emits structured // tracing events for direct connection attempts that bypass the proxy. #[cfg(target_os = "linux")] - let _bypass_monitor = if netns.is_some() { + let _bypass_monitor = netns.as_ref().and_then(|ns| { bypass_monitor::spawn( - netns.as_ref().expect("netns is Some").name().to_string(), + ns.name().to_string(), entrypoint_pid.clone(), bypass_denial_tx, ) - } else { - None - }; + }); // On non-Linux, bypass_denial_tx is unused (no /dev/kmsg). #[cfg(not(target_os = "linux"))] @@ -587,11 +596,11 @@ pub async fn run_sandbox( } match waitpid(pid, Some(WaitPidFlag::WNOHANG)) { - Ok(WaitStatus::StillAlive) | Err(nix::errno::Errno::ECHILD) => {} + Ok(WaitStatus::StillAlive) + | Err(nix::errno::Errno::ECHILD | nix::errno::Errno::EINTR) => {} Ok(reaped) => { tracing::debug!(?reaped, "Reaped orphaned child process"); } - Err(nix::errno::Errno::EINTR) => {} Err(e) => { tracing::debug!(error = %e, "waitpid error during orphan reap"); break; @@ -601,18 +610,12 @@ pub async fn run_sandbox( } }); - if let Some(listen_addr) = ssh_listen_addr { - let addr: SocketAddr = listen_addr.parse().into_diagnostic()?; + let ssh_socket_path: Option = ssh_socket_path.map(std::path::PathBuf::from); + if let Some(listen_path) = ssh_socket_path.clone() { let policy_clone = policy.clone(); let workdir_clone = workdir.clone(); - let secret = ssh_handshake_secret - .filter(|s| !s.is_empty()) - .ok_or_else(|| { - miette::miette!( - "OPENSHELL_SSH_HANDSHAKE_SECRET is required when SSH is enabled.\n\ - Set --ssh-handshake-secret or the OPENSHELL_SSH_HANDSHAKE_SECRET env var." - ) - })?; + let _ = ssh_handshake_secret; // retained in the signature for compat; unused + let _ = ssh_handshake_skew_secs; let proxy_url = ssh_proxy_url; let netns_fd = ssh_netns_fd; let ca_paths = ca_file_paths.clone(); @@ -622,12 +625,10 @@ pub async fn run_sandbox( tokio::spawn(async move { if let Err(err) = ssh::run_ssh_server( - addr, + listen_path, ssh_ready_tx, policy_clone, workdir_clone, - secret, - ssh_handshake_skew_secs, netns_fd, proxy_url, ca_paths, @@ -676,6 +677,18 @@ pub async fn run_sandbox( } } + // Spawn the persistent supervisor session if we have a gateway endpoint + // and sandbox identity. The session provides relay channels for SSH + // connect and ExecSandbox through the gateway. + if let (Some(endpoint), Some(id), Some(socket)) = ( + openshell_endpoint.as_ref(), + sandbox_id.as_ref(), + ssh_socket_path.as_ref(), + ) { + supervisor_session::spawn(endpoint.clone(), id.clone(), socket.clone()); + info!("supervisor session task spawned"); + } + #[cfg(target_os = "linux")] let mut handle = ProcessHandle::spawn( program, @@ -890,6 +903,9 @@ pub async fn run_sandbox( /// wins and the cluster bundle is not fetched. /// /// Returns `None` if neither source is configured (inference routing disabled). +// `routes`/`router` are intentionally distinct nouns (the route list vs the +// router that consumes them); both names are clearer than alternatives. +#[allow(clippy::similar_names)] async fn build_inference_context( sandbox_id: Option<&str>, openshell_endpoint: Option<&str>, @@ -1099,8 +1115,8 @@ pub(crate) fn bundle_to_resolved_routes( .routes .iter() .map(|r| { - let (auth, default_headers) = - openshell_core::inference::auth_for_provider_type(&r.provider_type); + let (auth, default_headers, passthrough_headers) = + openshell_core::inference::route_headers_for_provider_type(&r.provider_type); let timeout = if r.timeout_secs == 0 { openshell_router::config::DEFAULT_ROUTE_TIMEOUT } else { @@ -1114,6 +1130,7 @@ pub(crate) fn bundle_to_resolved_routes( protocols: r.protocols.clone(), auth, default_headers, + passthrough_headers, timeout, } }) @@ -1220,14 +1237,31 @@ const PROXY_BASELINE_READ_WRITE: &[&str] = &["/sandbox", "/tmp"]; /// socket at init time. If the directory exists but Landlock denies traversal /// (EACCES vs ECONNREFUSED), NVML returns `NVML_ERROR_INSUFFICIENT_PERMISSIONS` /// even though the daemon is optional. Only read/traversal access is needed. -const GPU_BASELINE_READ_ONLY: &[&str] = &["/run/nvidia-persistenced"]; +/// +/// `/usr/lib/wsl`: On WSL2, CDI bind-mounts GPU libraries (libdxcore.so, +/// libcuda.so.1.1, etc.) into paths under `/usr/lib/wsl/`. Although `/usr` +/// is already in `PROXY_BASELINE_READ_ONLY`, individual file bind-mounts may +/// not be covered by the parent-directory Landlock rule when the mount crosses +/// a filesystem boundary. Listing `/usr/lib/wsl` explicitly ensures traversal +/// is permitted regardless of Landlock's cross-mount behaviour. +const GPU_BASELINE_READ_ONLY: &[&str] = &[ + "/run/nvidia-persistenced", + "/usr/lib/wsl", // WSL2: CDI-injected GPU library directory +]; /// GPU read-write paths (static). /// /// `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvidia-uvm-tools`, -/// `/dev/nvidia-modeset`: control and UVM devices injected by CDI. -/// Landlock restricts `open(2)` on device files even when DAC allows it; -/// these need read-write because NVML/CUDA opens them with `O_RDWR`. +/// `/dev/nvidia-modeset`: control and UVM devices injected by CDI on native +/// Linux. Landlock restricts `open(2)` on device files even when DAC allows +/// it; these need read-write because NVML/CUDA opens them with `O_RDWR`. +/// These devices do not exist on WSL2 and will be skipped by the existence +/// check in `enrich_proto_baseline_paths()`. +/// +/// `/dev/dxg`: On WSL2, NVIDIA GPUs are exposed through the DXG kernel driver +/// (DirectX Graphics) rather than the native nvidia* devices. CDI injects +/// `/dev/dxg` as the sole GPU device node; it does not exist on native Linux +/// and will be skipped there by the existence check. /// /// `/proc`: CUDA writes to `/proc//task//comm` during `cuInit()` /// to set thread names. Without write access, `cuInit()` returns error 304. @@ -1241,12 +1275,17 @@ const GPU_BASELINE_READ_WRITE: &[&str] = &[ "/dev/nvidia-uvm", "/dev/nvidia-uvm-tools", "/dev/nvidia-modeset", + "/dev/dxg", // WSL2: DXG device (GPU via DirectX kernel driver, injected by CDI) "/proc", ]; /// Returns true if GPU devices are present in the container. +/// +/// Checks both the native Linux NVIDIA control device (`/dev/nvidiactl`) and +/// the WSL2 DXG device (`/dev/dxg`). CDI injects exactly one of these +/// depending on the host kernel; the other will not exist. fn has_gpu_devices() -> bool { - std::path::Path::new("/dev/nvidiactl").exists() + std::path::Path::new("/dev/nvidiactl").exists() || std::path::Path::new("/dev/dxg").exists() } /// Enumerate per-GPU device nodes (`/dev/nvidia0`, `/dev/nvidia1`, …). @@ -1333,17 +1372,18 @@ fn enrich_proto_baseline_paths(proto: &mut openshell_core::proto::SandboxPolicy) } } for path in &rw { - if !fs.read_write.iter().any(|p| p == path) { - if !std::path::Path::new(path).exists() { - debug!( - path, - "Baseline read-write path does not exist, skipping enrichment" - ); - continue; - } - fs.read_write.push(path.clone()); - modified = true; + if fs.read_only.iter().any(|p| p == path) || fs.read_write.iter().any(|p| p == path) { + continue; + } + if !std::path::Path::new(path).exists() { + debug!( + path, + "Baseline read-write path does not exist, skipping enrichment" + ); + continue; } + fs.read_write.push(path.clone()); + modified = true; } if modified { @@ -1387,17 +1427,18 @@ fn enrich_sandbox_baseline_paths(policy: &mut SandboxPolicy) { } for path in &rw { let p = std::path::PathBuf::from(path); - if !policy.filesystem.read_write.contains(&p) { - if !p.exists() { - debug!( - path, - "Baseline read-write path does not exist, skipping enrichment" - ); - continue; - } - policy.filesystem.read_write.push(p); - modified = true; + if policy.filesystem.read_only.contains(&p) || policy.filesystem.read_write.contains(&p) { + continue; + } + if !p.exists() { + debug!( + path, + "Baseline read-write path does not exist, skipping enrichment" + ); + continue; } + policy.filesystem.read_write.push(p); + modified = true; } if modified { @@ -1413,8 +1454,17 @@ fn enrich_sandbox_baseline_paths(policy: &mut SandboxPolicy) { } #[cfg(test)] +#[allow( + clippy::needless_raw_string_hashes, + clippy::iter_on_single_items, + clippy::similar_names, + clippy::manual_string_new, + clippy::doc_markdown, + reason = "Test code: test fixtures often use idiomatic forms not flagged in production." +)] mod baseline_tests { use super::*; + use crate::policy::{FilesystemPolicy, LandlockPolicy, ProcessPolicy}; #[test] fn proc_not_in_both_read_only_and_read_write_when_gpu_present() { @@ -1479,6 +1529,171 @@ mod baseline_tests { ); } } + + #[test] + fn proto_enrichment_preserves_explicit_read_only_for_baseline_read_write_paths() { + let mut policy = openshell_policy::restrictive_default_policy(); + policy.filesystem = Some(openshell_core::proto::FilesystemPolicy { + read_only: vec!["/tmp".to_string()], + read_write: vec![], + include_workdir: false, + }); + policy.network_policies.insert( + "test".into(), + openshell_core::proto::NetworkPolicyRule { + name: "test-rule".into(), + endpoints: vec![openshell_core::proto::NetworkEndpoint { + host: "example.com".into(), + port: 443, + ..Default::default() + }], + ..Default::default() + }, + ); + + enrich_proto_baseline_paths(&mut policy); + + let filesystem = policy.filesystem.expect("filesystem policy"); + assert!( + filesystem.read_only.contains(&"/tmp".to_string()), + "explicit read_only baseline path should be preserved" + ); + assert!( + !filesystem.read_write.contains(&"/tmp".to_string()), + "baseline enrichment must not promote explicit read_only /tmp to read_write" + ); + } + + #[test] + fn gpu_baseline_read_write_contains_dxg() { + // /dev/dxg must be present so WSL2 sandboxes get the Landlock + // read-write rule for the CDI-injected DXG device. The existence + // check in enrich_proto_baseline_paths() skips it on native Linux. + assert!( + GPU_BASELINE_READ_WRITE.contains(&"/dev/dxg"), + "/dev/dxg must be in GPU_BASELINE_READ_WRITE for WSL2 support" + ); + } + + #[test] + fn local_enrichment_preserves_explicit_read_only_for_baseline_read_write_paths() { + let mut policy = SandboxPolicy { + version: 1, + filesystem: FilesystemPolicy { + read_only: vec![std::path::PathBuf::from("/tmp")], + read_write: vec![], + include_workdir: false, + }, + network: NetworkPolicy { + mode: NetworkMode::Proxy, + proxy: Some(ProxyPolicy { http_addr: None }), + }, + landlock: LandlockPolicy::default(), + process: ProcessPolicy::default(), + }; + + enrich_sandbox_baseline_paths(&mut policy); + + assert!( + policy + .filesystem + .read_only + .contains(&std::path::PathBuf::from("/tmp")), + "explicit read_only baseline path should be preserved" + ); + assert!( + !policy + .filesystem + .read_write + .contains(&std::path::PathBuf::from("/tmp")), + "baseline enrichment must not promote explicit read_only /tmp to read_write" + ); + } + + #[test] + fn gpu_baseline_read_only_contains_usr_lib_wsl() { + // /usr/lib/wsl must be present so CDI-injected WSL2 GPU library + // bind-mounts are accessible under Landlock. Skipped on native Linux. + assert!( + GPU_BASELINE_READ_ONLY.contains(&"/usr/lib/wsl"), + "/usr/lib/wsl must be in GPU_BASELINE_READ_ONLY for WSL2 CDI library paths" + ); + } + + #[test] + fn has_gpu_devices_reflects_dxg_or_nvidiactl() { + // Verify the OR logic: result must match the manual disjunction of + // the two path checks. Passes in all environments. + let nvidiactl = std::path::Path::new("/dev/nvidiactl").exists(); + let dxg = std::path::Path::new("/dev/dxg").exists(); + assert_eq!( + has_gpu_devices(), + nvidiactl || dxg, + "has_gpu_devices() should be true iff /dev/nvidiactl or /dev/dxg exists" + ); + } +} + +/// Returns `true` if the error is transient and worth retrying. +/// +/// Walks the `miette::Report` error chain looking for a `tonic::Status`. If +/// found, only the gRPC codes that represent transient failures are retryable. +/// If no `tonic::Status` is present (e.g. a raw connection error), assume the +/// failure is transient. +fn is_retryable_error(err: &miette::Report) -> bool { + let mut source: Option<&dyn std::error::Error> = Some(err.as_ref()); + while let Some(e) = source { + if let Some(status) = e.downcast_ref::() { + return matches!( + status.code(), + tonic::Code::Unavailable + | tonic::Code::DeadlineExceeded + | tonic::Code::ResourceExhausted + | tonic::Code::Aborted + | tonic::Code::Internal + | tonic::Code::Unknown + ); + } + source = e.source(); + } + true +} + +/// Retry a gRPC operation with exponential backoff (capped at 4 s). +/// +/// Non-transient gRPC errors (e.g. `NOT_FOUND`, `INVALID_ARGUMENT`, +/// `PERMISSION_DENIED`) are returned immediately without retrying. +async fn grpc_retry(op_name: &str, f: F) -> Result +where + F: Fn() -> Fut, + Fut: Future>, +{ + let mut last_err = None; + for attempt in 1..=5u32 { + match f().await { + Ok(val) => return Ok(val), + Err(e) => { + if !is_retryable_error(&e) { + return Err(e); + } + if attempt < 5 { + warn!( + attempt, + max_attempts = 5, + error = %e, + "{op_name} failed, retrying" + ); + let backoff = Duration::from_secs((1u64 << (attempt - 1)).min(4)); + tokio::time::sleep(backoff).await; + } + last_err = Some(e); + } + } + } + Err(miette::miette!( + "{op_name} failed after 5 attempts: {}", + last_err.expect("loop executed at least once") + )) } /// Load sandbox policy from local files or gRPC. @@ -1541,54 +1756,54 @@ async fn load_policy( endpoint = %endpoint, "Fetching sandbox policy via gRPC" ); - let proto_policy = grpc_client::fetch_policy(endpoint, id).await?; + let proto_policy = + grpc_retry("Policy fetch", || grpc_client::fetch_policy(endpoint, id)).await?; - let mut proto_policy = match proto_policy { - Some(p) => p, - None => { - // No policy configured on the server. Discover from disk or - // fall back to the restrictive default, then sync to the - // gateway so it becomes the authoritative baseline. - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Other, "discovery") - .message("Server returned no policy; attempting local discovery") - .build() - ); - let mut discovered = discover_policy_from_disk_or_default(); - // Enrich before syncing so the gateway baseline includes - // baseline paths from the start. - enrich_proto_baseline_paths(&mut discovered); - let sandbox = sandbox.as_deref().ok_or_else(|| { - miette::miette!( - "Cannot sync discovered policy: sandbox not available.\n\ - Set OPENSHELL_SANDBOX or --sandbox to enable policy sync." - ) - })?; + let mut proto_policy = if let Some(p) = proto_policy { + p + } else { + // No policy configured on the server. Discover from disk or + // fall back to the restrictive default, then sync to the + // gateway so it becomes the authoritative baseline. + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Other, "discovery") + .message("Server returned no policy; attempting local discovery") + .build() + ); + let mut discovered = discover_policy_from_disk_or_default(); + // Enrich before syncing so the gateway baseline includes + // baseline paths from the start. + enrich_proto_baseline_paths(&mut discovered); + let sandbox = sandbox.as_deref().ok_or_else(|| { + miette::miette!( + "Cannot sync discovered policy: sandbox not available.\n\ + Set OPENSHELL_SANDBOX or --sandbox to enable policy sync." + ) + })?; - // Sync and re-fetch over a single connection to avoid extra - // TLS handshakes. - grpc_client::discover_and_sync_policy(endpoint, id, sandbox, &discovered).await? - } + // Sync and re-fetch over a single connection to avoid extra + // TLS handshakes. + grpc_retry("Policy discovery sync", || { + grpc_client::discover_and_sync_policy(endpoint, id, sandbox, &discovered) + }) + .await? }; // Ensure baseline filesystem paths are present for proxy-mode // sandboxes. If the policy was enriched, sync the updated version // back to the gateway so users can see the effective policy. let enriched = enrich_proto_baseline_paths(&mut proto_policy); - if enriched { - if let Some(sandbox_name) = sandbox.as_deref() { - if let Err(e) = - grpc_client::sync_policy(endpoint, sandbox_name, &proto_policy).await - { - warn!( - error = %e, - "Failed to sync enriched policy back to gateway (non-fatal)" - ); - } - } + if enriched + && let Some(sandbox_name) = sandbox.as_deref() + && let Err(e) = grpc_client::sync_policy(endpoint, sandbox_name, &proto_policy).await + { + warn!( + error = %e, + "Failed to sync enriched policy back to gateway (non-fatal)" + ); } // Build OPA engine from baked-in rules + typed proto data. @@ -1650,76 +1865,71 @@ fn discover_policy_from_path(path: &std::path::Path) -> openshell_core::proto::S parse_sandbox_policy, restrictive_default_policy, validate_sandbox_policy, }; - match std::fs::read_to_string(path) { - Ok(yaml) => { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") + let Ok(yaml) = std::fs::read_to_string(path) else { + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "default") + .message(format!( + "No policy file on disk, using restrictive default [path:{}]", + path.display() + )) + .build() + ); + return restrictive_default_policy(); + }; + ocsf_emit!( + ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .state(StateId::Enabled, "loaded") + .message(format!( + "Loaded sandbox policy from container disk [path:{}]", + path.display() + )) + .build() + ); + match parse_sandbox_policy(&yaml) { + Ok(policy) => { + // Validate the disk-loaded policy for safety. + if let Err(violations) = validate_sandbox_policy(&policy) { + let messages: Vec = violations.iter().map(ToString::to_string).collect(); + ocsf_emit!(DetectionFindingBuilder::new(ocsf_ctx()) + .activity(ActivityId::Open) + .severity(SeverityId::Medium) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .finding_info( + FindingInfo::new( + "unsafe-disk-policy", + "Unsafe Disk Policy Content", + ) + .with_desc(&format!( + "Disk policy at {} contains unsafe content: {}", + path.display(), + messages.join("; "), + )), + ) .message(format!( - "Loaded sandbox policy from container disk [path:{}]", + "Disk policy contains unsafe content, using restrictive default [path:{}]", path.display() )) - .build() - ); - match parse_sandbox_policy(&yaml) { - Ok(policy) => { - // Validate the disk-loaded policy for safety. - if let Err(violations) = validate_sandbox_policy(&policy) { - let messages: Vec = - violations.iter().map(ToString::to_string).collect(); - ocsf_emit!(DetectionFindingBuilder::new(ocsf_ctx()) - .activity(ActivityId::Open) - .severity(SeverityId::Medium) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .finding_info( - FindingInfo::new( - "unsafe-disk-policy", - "Unsafe Disk Policy Content", - ) - .with_desc(&format!( - "Disk policy at {} contains unsafe content: {}", - path.display(), - messages.join("; "), - )), - ) - .message(format!( - "Disk policy contains unsafe content, using restrictive default [path:{}]", - path.display() - )) - .build()); - return restrictive_default_policy(); - } - policy - } - Err(e) => { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Other, "fallback") - .message(format!( - "Failed to parse disk policy, using restrictive default [path:{} error:{e}]", - path.display() - )) - .build()); - restrictive_default_policy() - } + .build()); + return restrictive_default_policy(); } + policy } - Err(_) => { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "default") - .message(format!( - "No policy file on disk, using restrictive default [path:{}]", - path.display() - )) - .build() - ); + Err(e) => { + ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .state(StateId::Other, "fallback") + .message(format!( + "Failed to parse disk policy, using restrictive default [path:{} error:{e}]", + path.display() + )) + .build()); restrictive_default_policy() } } @@ -1884,7 +2094,7 @@ async fn flush_proposals_to_gateway( .map(|s| DenialSummary { sandbox_id: String::new(), host: s.host, - port: s.port as u32, + port: u32::from(s.port), binary: s.binary, ancestors: s.ancestors, deny_reason: s.deny_reason, @@ -2050,13 +2260,13 @@ async fn run_policy_poll_loop( .build() ); } - if result.version > 0 && result.policy_source == PolicySource::Sandbox { - if let Err(e) = client + if result.version > 0 + && result.policy_source == PolicySource::Sandbox + && let Err(e) = client .report_policy_status(sandbox_id, result.version, true, "") .await - { - warn!(error = %e, "Failed to report policy load success"); - } + { + warn!(error = %e, "Failed to report policy load success"); } } Err(e) => { @@ -2071,13 +2281,13 @@ async fn run_policy_poll_loop( result.version )) .build()); - if result.version > 0 && result.policy_source == PolicySource::Sandbox { - if let Err(report_err) = client + if result.version > 0 + && result.policy_source == PolicySource::Sandbox + && let Err(report_err) = client .report_policy_status(sandbox_id, result.version, false, &e.to_string()) .await - { - warn!(error = %report_err, "Failed to report policy load failure"); - } + { + warn!(error = %report_err, "Failed to report policy load failure"); } } } @@ -2129,8 +2339,8 @@ fn log_setting_changes( .status(StatusId::Success) .state(StateId::Enabled, "updated") .unmapped("key", serde_json::json!(key)) - .unmapped("old", serde_json::json!(old_val.to_string())) - .unmapped("new", serde_json::json!(new_val.to_string())) + .unmapped("old", serde_json::json!(old_val.clone())) + .unmapped("new", serde_json::json!(new_val.clone())) .message(format!( "Setting changed [key:{key} old:{old_val} new:{new_val}]" )) @@ -2145,7 +2355,7 @@ fn log_setting_changes( .status(StatusId::Success) .state(StateId::Enabled, "enabled") .unmapped("key", serde_json::json!(key)) - .unmapped("value", serde_json::json!(new_val.to_string())) + .unmapped("value", serde_json::json!(new_val.clone())) .message(format!("Setting added [key:{key} value:{new_val}]")) .build() ); @@ -2180,6 +2390,14 @@ fn format_setting_value(es: &openshell_core::proto::EffectiveSetting) -> String } #[cfg(test)] +#[allow( + clippy::needless_raw_string_hashes, + clippy::iter_on_single_items, + clippy::similar_names, + clippy::manual_string_new, + clippy::doc_markdown, + reason = "Test code: test fixtures often use idiomatic forms not flagged in production." +)] mod tests { use super::*; use crate::policy::{FilesystemPolicy, LandlockPolicy, ProcessPolicy}; @@ -2189,8 +2407,7 @@ mod tests { use std::os::unix::fs::{MetadataExt, symlink}; use temp_env::with_vars; - static ENV_LOCK: std::sync::LazyLock> = - std::sync::LazyLock::new(|| std::sync::Mutex::new(())); + static ENV_LOCK: LazyLock> = LazyLock::new(|| Mutex::new(())); #[test] fn bundle_to_resolved_routes_converts_all_fields() { @@ -2296,6 +2513,7 @@ mod tests { protocols: vec!["openai_chat_completions".to_string()], auth: openshell_core::inference::AuthHeader::Bearer, default_headers: vec![], + passthrough_headers: vec![], timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }, openshell_router::config::ResolvedRoute { @@ -2306,6 +2524,7 @@ mod tests { protocols: vec!["anthropic_messages".to_string()], auth: openshell_core::inference::AuthHeader::Custom("x-api-key"), default_headers: vec![], + passthrough_headers: vec![], timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }, ]; @@ -2595,6 +2814,7 @@ filesystem_policy: auth: openshell_core::inference::AuthHeader::Bearer, protocols: vec!["openai_chat_completions".to_string()], default_headers: vec![], + passthrough_headers: vec![], timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }]; diff --git a/crates/openshell-sandbox/src/log_push.rs b/crates/openshell-sandbox/src/log_push.rs index 17f9bcc3d7..8e053f79f9 100644 --- a/crates/openshell-sandbox/src/log_push.rs +++ b/crates/openshell-sandbox/src/log_push.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Push sandbox tracing events to the OpenShell server via gRPC. +//! Push sandbox tracing events to the `OpenShell` server via gRPC. //! //! A [`tracing`] layer captures log events and sends them through an mpsc //! channel to a background task. The task batches lines and streams them to @@ -15,7 +15,7 @@ use tracing::{Event, Subscriber}; use tracing_subscriber::Layer; use tracing_subscriber::layer::Context; -/// Tracing layer that pushes log events to the OpenShell server. +/// Tracing layer that pushes log events to the `OpenShell` server. /// /// Events are sent best-effort via `try_send` — if the channel is full the /// event is dropped. Logging must never block the sandbox. @@ -248,7 +248,7 @@ async fn drain_during_backoff( let deadline = tokio::time::Instant::now() + delay; loop { tokio::select! { - _ = tokio::time::sleep_until(deadline) => { return; } + () = tokio::time::sleep_until(deadline) => { return; } line = rx.recv() => { match line { Some(l) => { diff --git a/crates/openshell-sandbox/src/main.rs b/crates/openshell-sandbox/src/main.rs index a37dce0e4f..a62129582e 100644 --- a/crates/openshell-sandbox/src/main.rs +++ b/crates/openshell-sandbox/src/main.rs @@ -1,13 +1,13 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! OpenShell Sandbox - process sandbox and monitor. +//! `OpenShell` Sandbox - process sandbox and monitor. use std::sync::Arc; use std::sync::atomic::AtomicBool; use clap::Parser; -use miette::Result; +use miette::{IntoDiagnostic, Result}; use openshell_ocsf::{OcsfJsonlLayer, OcsfShorthandLayer}; use tracing::{info, warn}; use tracing_subscriber::EnvFilter; @@ -16,7 +16,7 @@ use tracing_subscriber::{Layer, layer::SubscriberExt, util::SubscriberInitExt}; use openshell_sandbox::run_sandbox; -/// OpenShell Sandbox - process isolation and monitoring. +/// `OpenShell` Sandbox - process isolation and monitoring. #[derive(Parser, Debug)] #[command(name = "openshell-sandbox")] #[command(version = openshell_core::VERSION)] @@ -40,7 +40,7 @@ struct Args { #[arg(long, short = 'i')] interactive: bool, - /// Sandbox ID for fetching policy via gRPC from OpenShell server. + /// Sandbox ID for fetching policy via gRPC from `OpenShell` server. /// Requires --openshell-endpoint to be set. #[arg(long, env = "OPENSHELL_SANDBOX_ID")] sandbox_id: Option, @@ -50,7 +50,7 @@ struct Args { #[arg(long, env = "OPENSHELL_SANDBOX")] sandbox: Option, - /// OpenShell server gRPC endpoint for fetching policy. + /// `OpenShell` server gRPC endpoint for fetching policy. /// Required when using --sandbox-id. #[arg(long, env = "OPENSHELL_ENDPOINT")] openshell_endpoint: Option, @@ -69,9 +69,11 @@ struct Args { #[arg(long, default_value = "warn", env = "OPENSHELL_LOG_LEVEL")] log_level: String, - /// SSH listen address for sandbox access. - #[arg(long, env = "OPENSHELL_SSH_LISTEN_ADDR")] - ssh_listen_addr: Option, + /// Filesystem path to the Unix socket the embedded SSH daemon binds. + /// The supervisor bridges `RelayStream` traffic from the gateway onto + /// this socket; nothing else should connect to it. + #[arg(long, env = "OPENSHELL_SSH_SOCKET_PATH")] + ssh_socket_path: Option, /// Shared secret for gateway-to-sandbox SSH handshake. #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SECRET")] @@ -96,11 +98,10 @@ struct Args { health_port: u16, } -#[tokio::main] -async fn main() -> Result<()> { +fn main() -> Result<()> { let args = Args::parse(); - // Try to open a rolling log file; fall back to stdout-only logging if it fails + // Try to open a rolling log file; fall back to stderr-only logging if it fails // (e.g., /var/log is not writable in custom workload images). // Rotates daily, keeps the 3 most recent files to bound disk usage. let file_logging = tracing_appender::rolling::RollingFileAppender::builder() @@ -115,119 +116,128 @@ async fn main() -> Result<()> { (writer, guard) }); - let stdout_filter = + let console_filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)); - // Install rustls crypto provider before any TLS connections (including log push). - let _ = rustls::crypto::ring::default_provider().install_default(); - - // Set up optional log push layer (gRPC mode only). - let log_push_state = if let (Some(sandbox_id), Some(endpoint)) = - (&args.sandbox_id, &args.openshell_endpoint) - { - let (tx, handle) = - openshell_sandbox::log_push::spawn_log_push_task(endpoint.clone(), sandbox_id.clone()); - let layer = openshell_sandbox::log_push::LogPushLayer::new(sandbox_id.clone(), tx); - Some((layer, handle)) - } else { - None - }; - let push_layer = log_push_state.as_ref().map(|(layer, _)| layer.clone()); - let _log_push_handle = log_push_state.map(|(_, handle)| handle); - - // Shared flag: the sandbox poll loop toggles this when the - // `ocsf_json_enabled` setting changes. The JSONL layer checks it - // on each event and short-circuits when false. - let ocsf_enabled = Arc::new(AtomicBool::new(false)); - - // Keep guards alive for the entire process. When a guard is dropped the - // non-blocking writer flushes remaining logs. - let (_file_guard, _jsonl_guard) = if let Some((file_writer, file_guard)) = file_logging { - let file_filter = EnvFilter::new("info"); - - // OCSF JSONL file: rolling appender matching the main log file - // (daily rotation, 3 files max). Created eagerly but gated by the - // enabled flag — no JSONL is written until ocsf_json_enabled is set. - let jsonl_logging = tracing_appender::rolling::RollingFileAppender::builder() - .rotation(tracing_appender::rolling::Rotation::DAILY) - .filename_prefix("openshell-ocsf") - .filename_suffix("log") - .max_log_files(3) - .build("/var/log") - .ok() - .map(|roller| { - let (writer, guard) = tracing_appender::non_blocking(roller); - let layer = OcsfJsonlLayer::new(writer).with_enabled_flag(ocsf_enabled.clone()); - (layer, guard) - }); - let (jsonl_layer, jsonl_guard) = match jsonl_logging { - Some((layer, guard)) => (Some(layer), Some(guard)), - None => (None, None), + let runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .into_diagnostic()?; + + let exit_code = runtime.block_on(async move { + // Install rustls crypto provider before any TLS connections (including log push). + let _ = rustls::crypto::ring::default_provider().install_default(); + + // Set up optional log push layer (gRPC mode only). + let log_push_state = if let (Some(sandbox_id), Some(endpoint)) = + (&args.sandbox_id, &args.openshell_endpoint) + { + let (tx, handle) = openshell_sandbox::log_push::spawn_log_push_task( + endpoint.clone(), + sandbox_id.clone(), + ); + let layer = openshell_sandbox::log_push::LogPushLayer::new(sandbox_id.clone(), tx); + Some((layer, handle)) + } else { + None + }; + let push_layer = log_push_state.as_ref().map(|(layer, _)| layer.clone()); + let _log_push_handle = log_push_state.map(|(_, handle)| handle); + + // Shared flag: the sandbox poll loop toggles this when the + // `ocsf_json_enabled` setting changes. The JSONL layer checks it + // on each event and short-circuits when false. + let ocsf_enabled = Arc::new(AtomicBool::new(false)); + + // Keep guards alive for the entire process. When a guard is dropped the + // non-blocking writer flushes remaining logs. + let (_file_guard, _jsonl_guard) = if let Some((file_writer, file_guard)) = file_logging { + let file_filter = EnvFilter::new("info"); + + // OCSF JSONL file: rolling appender matching the main log file + // (daily rotation, 3 files max). Created eagerly but gated by the + // enabled flag — no JSONL is written until ocsf_json_enabled is set. + let jsonl_logging = tracing_appender::rolling::RollingFileAppender::builder() + .rotation(tracing_appender::rolling::Rotation::DAILY) + .filename_prefix("openshell-ocsf") + .filename_suffix("log") + .max_log_files(3) + .build("/var/log") + .ok() + .map(|roller| { + let (writer, guard) = tracing_appender::non_blocking(roller); + let layer = OcsfJsonlLayer::new(writer).with_enabled_flag(ocsf_enabled.clone()); + (layer, guard) + }); + let (jsonl_layer, jsonl_guard) = match jsonl_logging { + Some((layer, guard)) => (Some(layer), Some(guard)), + None => (None, None), + }; + + tracing_subscriber::registry() + .with( + OcsfShorthandLayer::new(std::io::stderr()) + .with_non_ocsf(true) + .with_filter(console_filter), + ) + .with( + OcsfShorthandLayer::new(file_writer) + .with_non_ocsf(true) + .with_filter(file_filter), + ) + .with(jsonl_layer.with_filter(LevelFilter::INFO)) + .with(push_layer.clone()) + .init(); + (Some(file_guard), jsonl_guard) + } else { + tracing_subscriber::registry() + .with( + OcsfShorthandLayer::new(std::io::stderr()) + .with_non_ocsf(true) + .with_filter(console_filter), + ) + .with(push_layer) + .init(); + // Log the warning after the subscriber is initialized + warn!("Could not open /var/log for log rotation; using stderr-only logging"); + (None, None) + }; + + // Get command - either from CLI args, environment variable, or default to /bin/bash + let command = if !args.command.is_empty() { + args.command + } else if let Ok(c) = std::env::var("OPENSHELL_SANDBOX_COMMAND") { + // Simple shell-like splitting on whitespace + c.split_whitespace().map(String::from).collect() + } else { + vec!["/bin/bash".to_string()] }; - tracing_subscriber::registry() - .with( - OcsfShorthandLayer::new(std::io::stdout()) - .with_non_ocsf(true) - .with_filter(stdout_filter), - ) - .with( - OcsfShorthandLayer::new(file_writer) - .with_non_ocsf(true) - .with_filter(file_filter), - ) - .with(jsonl_layer.with_filter(LevelFilter::INFO)) - .with(push_layer.clone()) - .init(); - (Some(file_guard), jsonl_guard) - } else { - tracing_subscriber::registry() - .with( - OcsfShorthandLayer::new(std::io::stdout()) - .with_non_ocsf(true) - .with_filter(stdout_filter), - ) - .with(push_layer) - .init(); - // Log the warning after the subscriber is initialized - warn!("Could not open /var/log for log rotation; using stdout-only logging"); - (None, None) - }; - - // Get command - either from CLI args, environment variable, or default to /bin/bash - let command = if !args.command.is_empty() { - args.command - } else if let Ok(c) = std::env::var("OPENSHELL_SANDBOX_COMMAND") { - // Simple shell-like splitting on whitespace - c.split_whitespace().map(String::from).collect() - } else { - vec!["/bin/bash".to_string()] - }; - - info!(command = ?command, "Starting sandbox"); - // Note: "Starting sandbox" stays as plain info!() since the OCSF context - // is not yet initialized at this point (run_sandbox hasn't been called). - // The shorthand layer will render it in fallback format. - - let exit_code = run_sandbox( - command, - args.workdir, - args.timeout, - args.interactive, - args.sandbox_id, - args.sandbox, - args.openshell_endpoint, - args.policy_rules, - args.policy_data, - args.ssh_listen_addr, - args.ssh_handshake_secret, - args.ssh_handshake_skew_secs, - args.health_check, - args.health_port, - args.inference_routes, - ocsf_enabled, - ) - .await?; + info!(command = ?command, "Starting sandbox"); + // Note: "Starting sandbox" stays as plain info!() since the OCSF context + // is not yet initialized at this point (run_sandbox hasn't been called). + // The shorthand layer will render it in fallback format. + + run_sandbox( + command, + args.workdir, + args.timeout, + args.interactive, + args.sandbox_id, + args.sandbox, + args.openshell_endpoint, + args.policy_rules, + args.policy_data, + args.ssh_socket_path, + args.ssh_handshake_secret, + args.ssh_handshake_skew_secs, + args.health_check, + args.health_port, + args.inference_routes, + ocsf_enabled, + ) + .await + })?; std::process::exit(exit_code); } diff --git a/crates/openshell-sandbox/src/mechanistic_mapper.rs b/crates/openshell-sandbox/src/mechanistic_mapper.rs index 3cfdf3f541..cb6daa550f 100644 --- a/crates/openshell-sandbox/src/mechanistic_mapper.rs +++ b/crates/openshell-sandbox/src/mechanistic_mapper.rs @@ -178,13 +178,13 @@ pub async fn generate_proposals(summaries: &[DenialSummary]) -> Vec .map(|(_, name)| format!(" ({name})")) .unwrap_or_default(); - let private_ip_note = if !allowed_ips.is_empty() { + let private_ip_note = if allowed_ips.is_empty() { + String::new() + } else { format!( " Host resolves to private IP ({}); allowed_ips included for SSRF override.", allowed_ips.join(", ") ) - } else { - String::new() }; // Note: hit_count in the DB accumulates across flush cycles, so we @@ -226,7 +226,7 @@ pub async fn generate_proposals(summaries: &[DenialSummary]) -> Vec decided_at_ms: 0, stage, supersedes_chunk_id: String::new(), - hit_count: total_count as i32, + hit_count: total_count.cast_signed(), first_seen_ms, last_seen_ms, binary: binary.clone(), @@ -336,16 +336,15 @@ fn generate_security_notes(host: &str, port: u16, is_ssrf: bool) -> String { /// Falls back to the exact observed path when no pattern applies. fn build_l7_rules(samples: &HashMap<(String, String), u32>) -> Vec { // Deduplicate after generalisation. - let mut seen: HashMap<(String, String), ()> = HashMap::new(); + let mut seen: std::collections::HashSet<(String, String)> = std::collections::HashSet::new(); let mut rules = Vec::new(); for (method, path) in samples.keys() { let generalised = generalise_path(path); let key = (method.clone(), generalised.clone()); - if seen.contains_key(&key) { + if !seen.insert(key) { continue; } - seen.insert(key, ()); rules.push(L7Rule { allow: Some(L7Allow { @@ -353,6 +352,9 @@ fn build_l7_rules(samples: &HashMap<(String, String), u32>) -> Vec { path: generalised, command: String::new(), query: HashMap::new(), + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), }), }); } @@ -405,7 +407,7 @@ fn looks_like_id(segment: &str) -> bool { return true; } // UUID-ish (contains dashes, 32+ hex chars) - let hex_only: String = segment.chars().filter(|c| c.is_ascii_hexdigit()).collect(); + let hex_only: String = segment.chars().filter(char::is_ascii_hexdigit).collect(); if hex_only.len() >= 24 && segment.contains('-') { return true; } @@ -450,10 +452,11 @@ async fn resolve_allowed_ips_if_private(host: &str, port: u32) -> Vec { let addrs = match tokio::net::lookup_host(&addr).await { Ok(addrs) => addrs.collect::>(), Err(e) => { + let port_u16 = u16::try_from(port).unwrap_or(u16::MAX); let event = openshell_ocsf::NetworkActivityBuilder::new(crate::ocsf_ctx()) .activity(openshell_ocsf::ActivityId::Fail) .severity(openshell_ocsf::SeverityId::Low) - .dst_endpoint(openshell_ocsf::Endpoint::from_domain(host, port as u16)) + .dst_endpoint(openshell_ocsf::Endpoint::from_domain(host, port_u16)) .message(format!("DNS resolution failed for allowed_ips check: {e}")) .build(); openshell_ocsf::ocsf_emit!(event); @@ -462,10 +465,11 @@ async fn resolve_allowed_ips_if_private(host: &str, port: u32) -> Vec { }; if addrs.is_empty() { + let port_u16 = u16::try_from(port).unwrap_or(u16::MAX); let event = openshell_ocsf::NetworkActivityBuilder::new(crate::ocsf_ctx()) .activity(openshell_ocsf::ActivityId::Fail) .severity(openshell_ocsf::SeverityId::Low) - .dst_endpoint(openshell_ocsf::Endpoint::from_domain(host, port as u16)) + .dst_endpoint(openshell_ocsf::Endpoint::from_domain(host, port_u16)) .message(format!( "DNS resolution returned no addresses for {host}:{port}" )) diff --git a/crates/openshell-sandbox/src/opa.rs b/crates/openshell-sandbox/src/opa.rs index 0069bcc3ee..5897679a02 100644 --- a/crates/openshell-sandbox/src/opa.rs +++ b/crates/openshell-sandbox/src/opa.rs @@ -11,7 +11,10 @@ use crate::policy::{FilesystemPolicy, LandlockCompatibility, LandlockPolicy, Pro use miette::Result; use openshell_core::proto::SandboxPolicy as ProtoSandboxPolicy; use std::path::{Path, PathBuf}; -use std::sync::Mutex; +use std::sync::{ + Arc, Mutex, + atomic::{AtomicU64, Ordering}, +}; /// Baked-in rego rules for OPA policy evaluation. /// These rules define the network access decision logic and static config @@ -64,6 +67,68 @@ pub struct SandboxConfig { /// (one eval per CONNECT request). pub struct OpaEngine { engine: Mutex, + generation: Arc, +} + +/// Generation guard captured when an HTTP tunnel or request path starts. +#[derive(Clone)] +pub struct PolicyGenerationGuard { + captured_generation: u64, + current_generation: Arc, +} + +impl PolicyGenerationGuard { + pub fn captured_generation(&self) -> u64 { + self.captured_generation + } + + pub fn current_generation(&self) -> u64 { + self.current_generation.load(Ordering::Acquire) + } + + pub fn is_stale(&self) -> bool { + self.current_generation() != self.captured_generation + } + + pub fn ensure_current(&self) -> Result<()> { + if self.is_stale() { + return Err(miette::miette!( + "policy generation is stale [captured_generation:{} current_generation:{}]", + self.captured_generation(), + self.current_generation(), + )); + } + Ok(()) + } +} + +/// Per-tunnel L7 policy evaluator bound to the engine generation captured when +/// the tunnel was established. +pub struct TunnelPolicyEngine { + engine: Mutex, + generation_guard: PolicyGenerationGuard, +} + +impl TunnelPolicyEngine { + pub fn captured_generation(&self) -> u64 { + self.generation_guard.captured_generation() + } + + pub fn current_generation(&self) -> u64 { + self.generation_guard.current_generation() + } + + pub fn is_stale(&self) -> bool { + self.generation_guard.is_stale() + } + + pub fn generation_guard(&self) -> &PolicyGenerationGuard { + &self.generation_guard + } + + pub(crate) fn engine(&self) -> &Mutex { + &self.engine + } } impl OpaEngine { @@ -84,6 +149,7 @@ impl OpaEngine { .map_err(|e| miette::miette!("{e}"))?; Ok(Self { engine: Mutex::new(engine), + generation: Arc::new(AtomicU64::new(0)), }) } @@ -101,6 +167,7 @@ impl OpaEngine { .map_err(|e| miette::miette!("{e}"))?; Ok(Self { engine: Mutex::new(engine), + generation: Arc::new(AtomicU64::new(0)), }) } @@ -137,7 +204,7 @@ impl OpaEngine { .severity(openshell_ocsf::SeverityId::Medium) .status(openshell_ocsf::StatusId::Success) .state(openshell_ocsf::StateId::Enabled, "validated") - .unmapped("warning", serde_json::json!(w.to_string())) + .unmapped("warning", serde_json::json!(w.clone())) .message(format!("L7 policy validation warning: {w}")) .build() ); @@ -162,6 +229,7 @@ impl OpaEngine { .map_err(|e| miette::miette!("{e}"))?; Ok(Self { engine: Mutex::new(engine), + generation: Arc::new(AtomicU64::new(0)), }) } @@ -233,6 +301,14 @@ impl OpaEngine { /// Uses the OPA `network_action` rule which returns one of: /// `"allow"` or `"deny"`. pub fn evaluate_network_action(&self, input: &NetworkInput) -> Result { + Ok(self.evaluate_network_action_with_generation(input)?.0) + } + + /// Evaluate network action and return the policy generation used for the evaluation. + pub fn evaluate_network_action_with_generation( + &self, + input: &NetworkInput, + ) -> Result<(NetworkAction, u64)> { let ancestor_strs: Vec = input .ancestors .iter() @@ -259,6 +335,7 @@ impl OpaEngine { .engine .lock() .map_err(|_| miette::miette!("OPA engine lock poisoned"))?; + let generation = self.current_generation(); engine .set_input_json(&input_json.to_string()) @@ -278,15 +355,14 @@ impl OpaEngine { Some(value_to_string(&matched)) }; - match action_str.as_str() { - "allow" => Ok(NetworkAction::Allow { matched_policy }), - _ => { - let reason_val = engine - .eval_rule("data.openshell.sandbox.deny_reason".into()) - .map_err(|e| miette::miette!("{e}"))?; - let reason = value_to_string(&reason_val); - Ok(NetworkAction::Deny { reason }) - } + if action_str == "allow" { + Ok((NetworkAction::Allow { matched_policy }, generation)) + } else { + let reason_val = engine + .eval_rule("data.openshell.sandbox.deny_reason".into()) + .map_err(|e| miette::miette!("{e}"))?; + let reason = value_to_string(&reason_val); + Ok((NetworkAction::Deny { reason }, generation)) } } @@ -307,6 +383,7 @@ impl OpaEngine { .lock() .map_err(|_| miette::miette!("OPA engine lock poisoned"))?; *engine = new_engine; + self.generation.fetch_add(1, Ordering::AcqRel); Ok(()) } @@ -341,9 +418,29 @@ impl OpaEngine { .lock() .map_err(|_| miette::miette!("OPA engine lock poisoned"))?; *engine = new_engine; + self.generation.fetch_add(1, Ordering::AcqRel); Ok(()) } + /// Current policy generation. Successful reloads increment this value. + pub fn current_generation(&self) -> u64 { + self.generation.load(Ordering::Acquire) + } + + /// Return a guard for a previously captured policy generation. + pub fn generation_guard(&self, expected_generation: u64) -> Result { + let generation = self.current_generation(); + if generation != expected_generation { + return Err(miette::miette!( + "policy changed before HTTP relay started [expected_generation:{expected_generation} current_generation:{generation}]" + )); + } + Ok(PolicyGenerationGuard { + captured_generation: generation, + current_generation: Arc::clone(&self.generation), + }) + } + /// Query static sandbox configuration from the OPA data module. /// /// Extracts `filesystem_policy`, `landlock`, and `process` from the Rego @@ -386,6 +483,23 @@ impl OpaEngine { /// to get the full endpoint object for the matched policy. Returns the raw /// `regorus::Value` which can be parsed by `l7::parse_l7_config()`. pub fn query_endpoint_config(&self, input: &NetworkInput) -> Result> { + Ok(self.query_endpoint_config_with_generation(input)?.0) + } + + /// Query L7 endpoint config and return the policy generation used for the query. + pub fn query_endpoint_config_with_generation( + &self, + input: &NetworkInput, + ) -> Result<(Option, u64)> { + let (configs, generation) = self.query_endpoint_configs_with_generation(input)?; + Ok((configs.into_iter().next(), generation)) + } + + /// Query all matching endpoint configs and return the policy generation used for the query. + pub fn query_endpoint_configs_with_generation( + &self, + input: &NetworkInput, + ) -> Result<(Vec, u64)> { let ancestor_strs: Vec = input .ancestors .iter() @@ -412,19 +526,20 @@ impl OpaEngine { .engine .lock() .map_err(|_| miette::miette!("OPA engine lock poisoned"))?; + let generation = self.current_generation(); engine .set_input_json(&input_json.to_string()) .map_err(|e| miette::miette!("{e}"))?; let val = engine - .eval_rule("data.openshell.sandbox.matched_endpoint_config".into()) + .eval_rule("data.openshell.sandbox._matching_endpoint_configs".into()) .map_err(|e| miette::miette!("{e}"))?; - if val == regorus::Value::Undefined { - Ok(None) - } else { - Ok(Some(val)) + match val { + regorus::Value::Undefined => Ok((Vec::new(), generation)), + regorus::Value::Array(values) => Ok((values.to_vec(), generation)), + other => Ok((vec![other], generation)), } } @@ -435,10 +550,10 @@ impl OpaEngine { /// match. This is used by the proxy to decide between full SSRF blocking /// and allowlist-based IP validation. pub fn query_allowed_ips(&self, input: &NetworkInput) -> Result> { - match self.query_endpoint_config(input)? { - Some(val) => Ok(get_str_array(&val, "allowed_ips")), - None => Ok(vec![]), - } + Ok(self + .query_endpoint_config(input)? + .map(|val| get_str_array(&val, "allowed_ips")) + .unwrap_or_default()) } /// Clone the inner regorus engine for per-tunnel L7 evaluation. @@ -446,12 +561,24 @@ impl OpaEngine { /// With the `arc` feature enabled, this shares compiled policy via Arc /// and only duplicates interpreter state (~microseconds). The cloned /// engine can be used without Mutex contention. - pub fn clone_engine_for_tunnel(&self) -> Result { + pub fn clone_engine_for_tunnel(&self, expected_generation: u64) -> Result { let engine = self .engine .lock() .map_err(|_| miette::miette!("OPA engine lock poisoned"))?; - Ok(engine.clone()) + let generation = self.current_generation(); + if generation != expected_generation { + return Err(miette::miette!( + "policy changed before L7 tunnel started [expected_generation:{expected_generation} current_generation:{generation}]" + )); + } + Ok(TunnelPolicyEngine { + engine: Mutex::new(engine.clone()), + generation_guard: PolicyGenerationGuard { + captured_generation: generation, + current_generation: Arc::clone(&self.generation), + }, + }) } } @@ -557,7 +684,7 @@ fn preprocess_yaml_data(yaml_str: &str) -> Result { .severity(openshell_ocsf::SeverityId::Medium) .status(openshell_ocsf::StatusId::Success) .state(openshell_ocsf::StateId::Enabled, "validated") - .unmapped("warning", serde_json::json!(w.to_string())) + .unmapped("warning", serde_json::json!(w.clone())) .message(format!("L7 policy validation warning: {w}")) .build() ); @@ -594,9 +721,8 @@ fn normalize_endpoint_ports(data: &mut serde_json::Value) { }; for ep in endpoints.iter_mut() { - let ep_obj = match ep.as_object_mut() { - Some(obj) => obj, - None => continue, + let Some(ep_obj) = ep.as_object_mut() else { + continue; }; // If "ports" already exists and is non-empty, keep it. @@ -637,10 +763,12 @@ fn normalize_endpoint_ports(data: &mut serde_json::Value) { /// - Path is not a symlink /// - Resolution fails (binary doesn't exist in container) /// - Resolved path equals the original +/// /// Normalize a path by resolving `.` and `..` components without touching /// the filesystem. Only works correctly for absolute paths. -fn normalize_path(path: &std::path::Path) -> std::path::PathBuf { - let mut result = std::path::PathBuf::new(); +#[cfg(any(target_os = "linux", test))] +fn normalize_path(path: &Path) -> PathBuf { + let mut result = PathBuf::new(); for component in path.components() { match component { std::path::Component::ParentDir => { @@ -664,7 +792,7 @@ fn resolve_binary_in_container(policy_path: &str, entrypoint_pid: u32) -> Option // /proc//root itself (a kernel pseudo-symlink to /) which // strips the prefix we need. read_link only reads the target of // the specified symlink, keeping us in the container's namespace. - let mut resolved = std::path::PathBuf::from(policy_path); + let mut resolved = PathBuf::from(policy_path); // Linux SYMLOOP_MAX is 40; stop before infinite loops for _ in 0..40 { @@ -820,6 +948,9 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St vec![] }; let mut ep = serde_json::json!({"host": e.host, "ports": ports}); + if !e.path.is_empty() { + ep["path"] = e.path.clone().into(); + } if !e.protocol.is_empty() { ep["protocol"] = e.protocol.clone().into(); } @@ -842,7 +973,14 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St "method": a.map_or("", |a| &a.method), "path": a.map_or("", |a| &a.path), "command": a.map_or("", |a| &a.command), + "operation_type": a.map_or("", |a| &a.operation_type), + "operation_name": a.map_or("", |a| &a.operation_name), }); + if let Some(a) = a + && !a.fields.is_empty() + { + allow["fields"] = a.fields.clone().into(); + } let query: serde_json::Map = a .map(|allow| { allow @@ -889,6 +1027,15 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St if !d.command.is_empty() { deny["command"] = d.command.clone().into(); } + if !d.operation_type.is_empty() { + deny["operation_type"] = d.operation_type.clone().into(); + } + if !d.operation_name.is_empty() { + deny["operation_name"] = d.operation_name.clone().into(); + } + if !d.fields.is_empty() { + deny["fields"] = d.fields.clone().into(); + } let query: serde_json::Map = d .query .iter() @@ -911,6 +1058,32 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St .collect(); ep["deny_rules"] = deny_rules.into(); } + if e.allow_encoded_slash { + ep["allow_encoded_slash"] = true.into(); + } + if !e.persisted_queries.is_empty() { + ep["persisted_queries"] = e.persisted_queries.clone().into(); + } + if !e.graphql_persisted_queries.is_empty() { + let persisted: serde_json::Map = e + .graphql_persisted_queries + .iter() + .map(|(key, op)| { + ( + key.clone(), + serde_json::json!({ + "operation_type": op.operation_type, + "operation_name": op.operation_name, + "fields": op.fields, + }), + ) + }) + .collect(); + ep["graphql_persisted_queries"] = persisted.into(); + } + if e.graphql_max_body_bytes > 0 { + ep["graphql_max_body_bytes"] = e.graphql_max_body_bytes.into(); + } ep }) .collect(); @@ -946,6 +1119,13 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St } #[cfg(test)] +#[allow( + clippy::needless_raw_string_hashes, + clippy::similar_names, + clippy::doc_markdown, + clippy::match_wildcard_for_single_variants, + reason = "Test code: test fixtures and panic-on-unexpected matches are idiomatic in tests." +)] mod tests { use super::*; @@ -1595,6 +1775,42 @@ network_policies: any: ["foo-*", "bar-*"] binaries: - { path: /usr/bin/curl } + graphql_api: + name: graphql_api + endpoints: + - host: api.graphql.com + port: 443 + protocol: graphql + enforcement: enforce + persisted_queries: allow_registered + graphql_persisted_queries: + abc123: + operation_type: query + operation_name: Viewer + fields: [viewer] + rules: + - allow: + operation_type: query + fields: [viewer, repository] + - allow: + operation_type: mutation + operation_name: Issue* + fields: [createIssue, deleteRepository] + deny_rules: + - operation_type: mutation + fields: [deleteRepository] + binaries: + - { path: /usr/bin/curl } + graphql_readonly: + name: graphql_readonly + endpoints: + - host: gql.readonly.com + port: 443 + protocol: graphql + enforcement: enforce + access: read-only + binaries: + - { path: /usr/bin/curl } l4_only: name: l4_only endpoints: @@ -1642,6 +1858,45 @@ process: }) } + fn l7_graphql_input(host: &str, operations: serde_json::Value) -> serde_json::Value { + serde_json::json!({ + "network": { "host": host, "port": 443 }, + "exec": { + "path": "/usr/bin/curl", + "ancestors": [], + "cmdline_paths": [] + }, + "request": { + "method": "POST", + "path": "/graphql", + "query_params": {}, + "graphql": { + "operations": operations + } + } + }) + } + + fn l7_graphql_error_input(host: &str, error: &str) -> serde_json::Value { + serde_json::json!({ + "network": { "host": host, "port": 443 }, + "exec": { + "path": "/usr/bin/curl", + "ancestors": [], + "cmdline_paths": [] + }, + "request": { + "method": "POST", + "path": "/graphql", + "query_params": {}, + "graphql": { + "operations": [], + "error": error + } + } + }) + } + fn eval_l7(engine: &OpaEngine, input: &serde_json::Value) -> bool { let mut eng = engine.engine.lock().unwrap(); eng.set_input_json(&input.to_string()).unwrap(); @@ -1726,6 +1981,215 @@ process: } } + #[test] + fn l7_graphql_query_allowed_by_field_rule() { + let engine = l7_engine(); + let input = l7_graphql_input( + "api.graphql.com", + serde_json::json!([{ + "operation_type": "query", + "operation_name": "RepoLookup", + "fields": ["repository"], + "persisted_query": false + }]), + ); + assert!(eval_l7(&engine, &input)); + } + + #[test] + fn l7_graphql_unlisted_field_denied() { + let engine = l7_engine(); + let input = l7_graphql_input( + "api.graphql.com", + serde_json::json!([{ + "operation_type": "query", + "fields": ["viewer", "adminAuditLog"], + "persisted_query": false + }]), + ); + assert!(!eval_l7(&engine, &input)); + } + + #[test] + fn l7_graphql_batch_denied_if_any_operation_unallowed() { + let engine = l7_engine(); + let input = l7_graphql_input( + "api.graphql.com", + serde_json::json!([ + { + "operation_type": "query", + "fields": ["viewer"], + "persisted_query": false + }, + { + "operation_type": "mutation", + "operation_name": "DeleteRepo", + "fields": ["deleteRepository"], + "persisted_query": false + } + ]), + ); + assert!(!eval_l7(&engine, &input)); + } + + #[test] + fn l7_graphql_deny_rule_takes_precedence() { + let engine = l7_engine(); + let input = l7_graphql_input( + "api.graphql.com", + serde_json::json!([{ + "operation_type": "mutation", + "operation_name": "IssueDelete", + "fields": ["deleteRepository"], + "persisted_query": false + }]), + ); + assert!(!eval_l7(&engine, &input)); + } + + #[test] + fn l7_graphql_registered_hash_only_query_allowed() { + let engine = l7_engine(); + let input = l7_graphql_input( + "api.graphql.com", + serde_json::json!([{ + "operation_type": "", + "operation_name": "Viewer", + "fields": [], + "persisted_query": true, + "persisted_query_hash": "abc123" + }]), + ); + assert!(eval_l7(&engine, &input)); + } + + #[test] + fn l7_graphql_unregistered_hash_only_query_denied() { + let engine = l7_engine(); + let input = l7_graphql_input( + "api.graphql.com", + serde_json::json!([{ + "operation_type": "", + "operation_name": "Viewer", + "fields": [], + "persisted_query": true, + "persisted_query_hash": "missing" + }]), + ); + assert!(!eval_l7(&engine, &input)); + } + + #[test] + fn l7_graphql_unregistered_hash_only_query_has_deny_reason() { + let engine = l7_engine(); + let input = l7_graphql_input( + "api.graphql.com", + serde_json::json!([{ + "operation_type": "", + "operation_name": "Viewer", + "fields": [], + "persisted_query": true, + "persisted_query_hash": "missing" + }]), + ); + let mut eng = engine.engine.lock().unwrap(); + eng.set_input_json(&input.to_string()).unwrap(); + let val = eng + .eval_rule("data.openshell.sandbox.request_deny_reason".into()) + .unwrap(); + assert_eq!( + val, + regorus::Value::String("GraphQL persisted query is not registered".into()) + ); + } + + #[test] + fn l7_graphql_parse_error_denied() { + let engine = l7_engine(); + let input = l7_graphql_error_input("api.graphql.com", "GraphQL document parse error"); + assert!(!eval_l7(&engine, &input)); + } + + #[test] + fn l7_graphql_readonly_access_allows_query_and_denies_mutation() { + let engine = l7_engine(); + let query = l7_graphql_input( + "gql.readonly.com", + serde_json::json!([{ + "operation_type": "query", + "fields": ["viewer"], + "persisted_query": false + }]), + ); + assert!(eval_l7(&engine, &query)); + + let mutation = l7_graphql_input( + "gql.readonly.com", + serde_json::json!([{ + "operation_type": "mutation", + "fields": ["createIssue"], + "persisted_query": false + }]), + ); + assert!(!eval_l7(&engine, &mutation)); + } + + #[test] + fn l7_endpoint_path_scopes_rest_and_graphql_on_same_host() { + let data = r#" +network_policies: + mixed_api: + name: mixed_api + endpoints: + - host: api.github.test + port: 443 + path: "/repos/**" + protocol: rest + enforcement: enforce + rules: + - allow: + method: "*" + path: "/**" + - host: api.github.test + port: 443 + path: "/graphql" + protocol: graphql + enforcement: enforce + rules: + - allow: + operation_type: query + binaries: + - { path: /usr/bin/curl } +"#; + let engine = OpaEngine::from_strings(TEST_POLICY, data).unwrap(); + + let rest_write = l7_input("api.github.test", 443, "POST", "/repos/org/repo/issues"); + assert!(eval_l7(&engine, &rest_write)); + + let graphql_query = l7_graphql_input( + "api.github.test", + serde_json::json!([{ + "operation_type": "query", + "fields": ["viewer"], + "persisted_query": false + }]), + ); + assert!(eval_l7(&engine, &graphql_query)); + + let graphql_mutation = l7_graphql_input( + "api.github.test", + serde_json::json!([{ + "operation_type": "mutation", + "fields": ["deleteRepository"], + "persisted_query": false + }]), + ); + assert!( + !eval_l7(&engine, &graphql_mutation), + "REST rules on the same host must not allow a GraphQL mutation" + ); + } + #[test] fn l7_method_matching_case_insensitive() { let engine = l7_engine(); @@ -1827,6 +2291,9 @@ process: path: "/download".to_string(), command: String::new(), query, + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), }), }], ..Default::default() @@ -1939,6 +2406,63 @@ process: assert_eq!(l7.enforcement, crate::l7::EnforcementMode::Enforce); } + #[test] + fn l7_endpoint_config_preserves_proto_allow_encoded_slash() { + let mut network_policies = std::collections::HashMap::new(); + network_policies.insert( + "npm".to_string(), + NetworkPolicyRule { + name: "npm".to_string(), + endpoints: vec![NetworkEndpoint { + host: "registry.npmjs.org".to_string(), + port: 443, + protocol: "rest".to_string(), + enforcement: "enforce".to_string(), + access: "read-only".to_string(), + allow_encoded_slash: true, + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/node".to_string(), + ..Default::default() + }], + }, + ); + let proto = ProtoSandboxPolicy { + version: 1, + filesystem: Some(ProtoFs { + include_workdir: true, + read_only: vec![], + read_write: vec![], + }), + landlock: Some(openshell_core::proto::LandlockPolicy { + compatibility: "best_effort".to_string(), + }), + process: Some(ProtoProc { + run_as_user: "sandbox".to_string(), + run_as_group: "sandbox".to_string(), + }), + network_policies, + }; + + let engine = OpaEngine::from_proto(&proto).expect("engine from proto"); + let input = NetworkInput { + host: "registry.npmjs.org".into(), + port: 443, + binary_path: PathBuf::from("/usr/bin/node"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + + let config = engine + .query_endpoint_config(&input) + .unwrap() + .expect("endpoint config"); + let l7 = crate::l7::parse_l7_config(&config).unwrap(); + assert!(l7.allow_encoded_slash); + } + #[test] fn l7_endpoint_config_none_for_l4_only() { let engine = l7_engine(); @@ -1960,10 +2484,54 @@ process: #[test] fn l7_clone_engine_for_tunnel() { let engine = l7_engine(); - let cloned = engine.clone_engine_for_tunnel().unwrap(); + let cloned = engine + .clone_engine_for_tunnel(engine.current_generation()) + .unwrap(); // Verify the cloned engine can evaluate let input_json = l7_input("api.example.com", 8080, "GET", "/repos/myorg/foo"); - let mut eng = cloned; + let mut eng = cloned.engine().lock().unwrap(); + eng.set_input_json(&input_json.to_string()).unwrap(); + let val = eng + .eval_rule("data.openshell.sandbox.allow_request".into()) + .unwrap(); + assert_eq!(val, regorus::Value::from(true)); + } + + #[test] + fn policy_generation_starts_at_zero_and_increments_on_successful_reload() { + let engine = l7_engine(); + assert_eq!(engine.current_generation(), 0); + + engine.reload(TEST_POLICY, L7_TEST_DATA).unwrap(); + + assert_eq!(engine.current_generation(), 1); + } + + #[test] + fn policy_generation_does_not_increment_on_failed_reload() { + let engine = l7_engine(); + engine.reload(TEST_POLICY, L7_TEST_DATA).unwrap(); + assert_eq!(engine.current_generation(), 1); + + let invalid_l7_data = r#" +network_policies: + bad_api: + name: bad_api + endpoints: + - host: api.example.com + port: 8080 + protocol: invalid-protocol + binaries: + - { path: /usr/bin/curl } +"#; + assert!(engine.reload(TEST_POLICY, invalid_l7_data).is_err()); + assert_eq!(engine.current_generation(), 1); + + let input_json = l7_input("api.example.com", 8080, "GET", "/repos/myorg/foo"); + let cloned = engine + .clone_engine_for_tunnel(engine.current_generation()) + .unwrap(); + let mut eng = cloned.engine().lock().unwrap(); eng.set_input_json(&input_json.to_string()).unwrap(); let val = eng .eval_rule("data.openshell.sandbox.allow_request".into()) @@ -1971,6 +2539,43 @@ process: assert_eq!(val, regorus::Value::from(true)); } + #[test] + fn endpoint_config_generation_matches_query_generation() { + let engine = l7_engine(); + let input = NetworkInput { + host: "api.example.com".into(), + port: 8080, + binary_path: PathBuf::from("/usr/bin/curl"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + + let (config, generation) = engine + .query_endpoint_config_with_generation(&input) + .unwrap(); + assert!(config.is_some()); + assert_eq!(generation, engine.current_generation()); + + engine.reload(TEST_POLICY, L7_TEST_DATA).unwrap(); + + let (config, generation) = engine + .query_endpoint_config_with_generation(&input) + .unwrap(); + assert!(config.is_some()); + assert_eq!(generation, engine.current_generation()); + assert_eq!(generation, 1); + } + + #[test] + fn tunnel_clone_rejects_stale_generation() { + let engine = l7_engine(); + let captured_generation = engine.current_generation(); + engine.reload(TEST_POLICY, L7_TEST_DATA).unwrap(); + + assert!(engine.clone_engine_for_tunnel(captured_generation).is_err()); + } + // ======================================================================== // Deny rules tests // ======================================================================== @@ -3673,9 +4278,8 @@ network_policies: #[cfg(target_os = "linux")] fn procfs_root_accessible() -> bool { use std::os::unix::fs::symlink; - let dir = match tempfile::tempdir() { - Ok(d) => d, - Err(_) => return false, + let Ok(dir) = tempfile::tempdir() else { + return false; }; let target = dir.path().join("probe_target"); let link = dir.path().join("probe_link"); @@ -3694,6 +4298,8 @@ network_policies: #[cfg(target_os = "linux")] #[test] fn resolve_binary_with_real_symlink() { + use std::os::unix::fs::symlink; + if !procfs_root_accessible() { eprintln!("Skipping: /proc//root/ not accessible in this environment"); return; @@ -3701,7 +4307,6 @@ network_policies: // Create a real symlink in a temp directory and verify resolution // works through /proc/self/root (which maps to / on the host) - use std::os::unix::fs::symlink; let dir = tempfile::tempdir().unwrap(); let target = dir.path().join("python3.11"); let link = dir.path().join("python3"); @@ -3730,13 +4335,14 @@ network_policies: #[cfg(target_os = "linux")] #[test] fn resolve_binary_non_symlink_returns_none() { + use std::io::Write; + if !procfs_root_accessible() { eprintln!("Skipping: /proc//root/ not accessible in this environment"); return; } // A regular file should return None (no expansion needed) - use std::io::Write; let mut tmp = tempfile::NamedTempFile::new().unwrap(); tmp.write_all(b"regular file").unwrap(); tmp.flush().unwrap(); @@ -3754,13 +4360,14 @@ network_policies: #[cfg(target_os = "linux")] #[test] fn resolve_binary_multi_level_symlink() { + use std::os::unix::fs::symlink; + if !procfs_root_accessible() { eprintln!("Skipping: /proc//root/ not accessible in this environment"); return; } // Test multi-level symlink resolution: python3 -> python3.11 -> cpython3.11 - use std::os::unix::fs::symlink; let dir = tempfile::tempdir().unwrap(); let final_target = dir.path().join("cpython3.11"); let mid_link = dir.path().join("python3.11"); @@ -3785,6 +4392,8 @@ network_policies: #[cfg(target_os = "linux")] #[test] fn from_proto_with_pid_expands_symlinks_in_container() { + use std::os::unix::fs::symlink; + if !procfs_root_accessible() { eprintln!("Skipping: /proc//root/ not accessible in this environment"); return; @@ -3792,7 +4401,6 @@ network_policies: // End-to-end test: create a symlink, build engine with our PID, // verify the resolved path is allowed - use std::os::unix::fs::symlink; let dir = tempfile::tempdir().unwrap(); let target = dir.path().join("node22"); let link = dir.path().join("node"); @@ -3861,6 +4469,8 @@ network_policies: #[cfg(target_os = "linux")] #[test] fn reload_from_proto_with_pid_resolves_symlinks() { + use std::os::unix::fs::symlink; + if !procfs_root_accessible() { eprintln!("Skipping: /proc//root/ not accessible in this environment"); return; @@ -3868,7 +4478,6 @@ network_policies: // Test hot-reload path: initial engine at pid=0, then reload with // real PID to trigger symlink resolution - use std::os::unix::fs::symlink; let dir = tempfile::tempdir().unwrap(); let target = dir.path().join("python3.11"); let link = dir.path().join("python3"); diff --git a/crates/openshell-sandbox/src/process.rs b/crates/openshell-sandbox/src/process.rs index 4e92259294..0dc5138366 100644 --- a/crates/openshell-sandbox/src/process.rs +++ b/crates/openshell-sandbox/src/process.rs @@ -35,13 +35,13 @@ fn scrub_sensitive_env(cmd: &mut Command) { } #[cfg(unix)] -#[allow(unsafe_code)] -pub(crate) fn harden_child_process() -> Result<()> { +#[allow(unsafe_code, clippy::borrow_as_ptr)] +pub fn harden_child_process() -> Result<()> { let core_limit = libc::rlimit { rlim_cur: 0, rlim_max: 0, }; - let rc = unsafe { libc::setrlimit(libc::RLIMIT_CORE, &core_limit) }; + let rc = unsafe { libc::setrlimit(libc::RLIMIT_CORE, &raw const core_limit) }; if rc != 0 { return Err(miette::miette!( "Failed to disable core dumps: {}", @@ -49,6 +49,22 @@ pub(crate) fn harden_child_process() -> Result<()> { )); } + // Limit process creation to prevent fork bombs. 512 processes per UID is + // sufficient for typical agent workloads (shell, compilers, language servers) + // while preventing runaway forking. Set as a hard limit so the sandbox user + // cannot raise it after privilege drop. + let nproc_limit = libc::rlimit { + rlim_cur: 512, + rlim_max: 512, + }; + let rc = unsafe { libc::setrlimit(libc::RLIMIT_NPROC, &raw const nproc_limit) }; + if rc != 0 { + return Err(miette::miette!( + "Failed to set RLIMIT_NPROC: {}", + std::io::Error::last_os_error() + )); + } + #[cfg(target_os = "linux")] { let rc = unsafe { libc::prctl(libc::PR_SET_DUMPABLE, 0, 0, 0, 0) }; @@ -411,7 +427,10 @@ impl Drop for ProcessHandle { } } +// `effective_gid`/`effective_uid` are intentionally parallel names (same role +// for different identifiers) and the noise from renaming would obscure intent. #[cfg(unix)] +#[allow(clippy::similar_names)] pub fn drop_privileges(policy: &SandboxPolicy) -> Result<()> { let user_name = match policy.process.run_as_user.as_deref() { Some(name) if !name.is_empty() => Some(name), @@ -595,6 +614,17 @@ mod tests { } } + /// Unknown names may yield `Ok(None)` (`… not found …`) or `Err` when NSS fails first + /// (e.g. `ENOENT: No such file or directory`). + fn assert_unknown_identity_lookup_failed(msg: &str) { + assert!( + msg.contains("not found") + || msg.contains("ENOENT") + || msg.contains("No such file or directory"), + "expected unknown user/group lookup failure (…not found… or ENOENT): {msg}" + ); + } + #[test] fn drop_privileges_noop_when_no_user_or_group() { let policy = policy_with_process(ProcessPolicy { @@ -677,10 +707,7 @@ mod tests { let result = drop_privileges(&policy); assert!(result.is_err()); let msg = format!("{}", result.unwrap_err()); - assert!( - msg.contains("not found"), - "expected 'not found' in error: {msg}" - ); + assert_unknown_identity_lookup_failed(&msg); } #[test] @@ -693,10 +720,7 @@ mod tests { let result = drop_privileges(&policy); assert!(result.is_err()); let msg = format!("{}", result.unwrap_err()); - assert!( - msg.contains("not found"), - "expected 'not found' in error: {msg}" - ); + assert_unknown_identity_lookup_failed(&msg); } #[cfg(unix)] @@ -724,11 +748,7 @@ mod tests { let written = unsafe { libc::write(fds[1], bytes.as_ptr().cast(), bytes.len()) }; unsafe { libc::close(fds[1]); - libc::_exit(if written == bytes.len() as isize { - 0 - } else { - 1 - }); + libc::_exit(i32::from(written != bytes.len().cast_signed())); } } ForkResult::Parent { child } => { @@ -737,7 +757,7 @@ mod tests { let read = unsafe { libc::read(fds[0], bytes.as_mut_ptr().cast(), bytes.len()) }; unsafe { libc::close(fds[0]) }; assert_eq!( - read as usize, + read.cast_unsigned(), bytes.len(), "expected {} probe bytes, got {}", bytes.len(), @@ -775,7 +795,7 @@ mod tests { #[cfg(target_os = "linux")] #[allow(unsafe_code)] unsafe fn dumpable_flag_probe() -> i64 { - unsafe { libc::prctl(libc::PR_GET_DUMPABLE, 0, 0, 0, 0) as i64 } + unsafe { i64::from(libc::prctl(libc::PR_GET_DUMPABLE, 0, 0, 0, 0)) } } #[test] diff --git a/crates/openshell-sandbox/src/procfs.rs b/crates/openshell-sandbox/src/procfs.rs index 785a9489ec..988ee24120 100644 --- a/crates/openshell-sandbox/src/procfs.rs +++ b/crates/openshell-sandbox/src/procfs.rs @@ -7,11 +7,46 @@ //! for process-identity binding in the OPA proxy policy engine. use miette::Result; +#[cfg(target_os = "linux")] +use std::collections::HashSet; use std::path::Path; #[cfg(target_os = "linux")] use std::path::PathBuf; use tracing::debug; +/// Where a socket owner was discovered while scanning `/proc`. +#[cfg(target_os = "linux")] +#[derive(Clone, Debug, Eq, PartialEq)] +pub enum SocketOwnerSource { + /// Owner was found in the entrypoint process tree at the given BFS depth. + Descendant { depth: usize }, + /// Owner was found by scanning all of `/proc` after the descendant scan. + ProcFallback, +} + +/// A process with an fd pointing at a target socket inode. +#[cfg(target_os = "linux")] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct SocketOwner { + pub pid: u32, + pub source: SocketOwnerSource, +} + +/// All process owners for a TCP peer socket. +#[cfg(target_os = "linux")] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct TcpPeerSocketOwners { + pub inode: u64, + pub owners: Vec, +} + +#[cfg(target_os = "linux")] +#[derive(Clone, Debug, Eq, PartialEq)] +struct DescendantPid { + pid: u32, + depth: usize, +} + /// Read the binary path of a process via `/proc/{pid}/exe` symlink. /// /// Returns the canonical path to the executable that the process is running. @@ -19,17 +54,63 @@ use tracing::debug; /// `/proc/{pid}/cmdline` because `argv[0]` is trivially spoofable by any /// process and must not be used as a trusted identity source. /// -/// If this fails, ensure the proxy process has permission to read -/// `/proc//exe` (e.g. same user, or `CAP_SYS_PTRACE`). +/// ### Unlinked binaries (`(deleted)` suffix) +/// +/// When a running binary is unlinked from its filesystem path — the common +/// case is a `docker cp` hot-swap of `/opt/openshell/bin/openshell-sandbox` +/// during a `cluster-deploy-fast` dev upgrade — the kernel appends the +/// literal string `" (deleted)"` to the `/proc//exe` readlink target. +/// The raw tainted path (e.g. `"/opt/openshell/bin/openshell-sandbox (deleted)"`) +/// is not a real filesystem path: any downstream `stat()` fails with `ENOENT`. +/// +/// We strip the suffix so callers see a clean, grep-friendly path suitable +/// for cache keys and log messages. The strip is guarded: we only strip when +/// `stat()` on the raw readlink target reports `NotFound`, so a live executable +/// whose basename literally ends with `" (deleted)"` is returned unchanged. +/// The comparison is done on raw bytes via `OsStrExt`, so filenames that are +/// not valid UTF-8 are still handled correctly. Exactly one kernel-added +/// suffix is stripped. +/// +/// This does NOT claim the file at the stripped path is the same binary that +/// the process is executing — the on-disk inode may now be arbitrary. Callers +/// that need to verify the running binary's *contents* (for integrity +/// checking) should read the magic `/proc//exe` symlink directly via +/// `File::open`, which procfs resolves to the live in-memory executable even +/// when the original inode has been unlinked. +/// +/// If the readlink itself fails, ensure the proxy process has permission +/// to read `/proc//exe` (e.g. same user, or `CAP_SYS_PTRACE`). #[cfg(target_os = "linux")] pub fn binary_path(pid: i32) -> Result { - std::fs::read_link(format!("/proc/{pid}/exe")).map_err(|e| { + use std::ffi::OsString; + use std::io::ErrorKind; + use std::os::unix::ffi::{OsStrExt, OsStringExt}; + + const DELETED_SUFFIX: &[u8] = b" (deleted)"; + + let link = format!("/proc/{pid}/exe"); + let target = std::fs::read_link(&link).map_err(|e| { miette::miette!( "Failed to read /proc/{pid}/exe: {e}. \ Cannot determine binary identity — denying request. \ Hint: the proxy may need CAP_SYS_PTRACE or to run as the same user." ) - }) + })?; + + // Only strip when the raw readlink target cannot be stat'd and its bytes + // end with the kernel-added suffix. This preserves live executables whose + // basename legitimately ends with " (deleted)" and handles non-UTF-8 + // filenames correctly. + let raw_target_missing = + matches!(std::fs::metadata(&target), Err(err) if err.kind() == ErrorKind::NotFound); + + let bytes = target.as_os_str().as_bytes(); + if raw_target_missing && bytes.ends_with(DELETED_SUFFIX) { + let stripped = bytes[..bytes.len() - DELETED_SUFFIX.len()].to_vec(); + return Ok(PathBuf::from(OsString::from_vec(stripped))); + } + + Ok(target) } /// Resolve the binary path of the TCP peer inside a sandbox network namespace. @@ -39,20 +120,54 @@ pub fn binary_path(pid: i32) -> Result { /// that socket, and finally reads `/proc//exe` to get the binary path. #[cfg(target_os = "linux")] pub fn resolve_tcp_peer_binary(entrypoint_pid: u32, peer_port: u16) -> Result { + let owner = resolve_single_tcp_peer_owner(entrypoint_pid, peer_port)?; + binary_path(owner.pid.cast_signed()) +} + +/// Resolve all process owners for the TCP peer inside a sandbox network namespace. +/// +/// Multiple processes can legitimately hold the same socket inode after `fork()` +/// or fd passing. Callers that make security decisions must evaluate the full +/// owner set instead of selecting the first PID returned by `/proc` traversal. +#[cfg(target_os = "linux")] +pub fn resolve_tcp_peer_socket_owners( + entrypoint_pid: u32, + peer_port: u16, +) -> Result { let inode = parse_proc_net_tcp(entrypoint_pid, peer_port)?; - let pid = find_pid_by_socket_inode(inode, entrypoint_pid)?; - binary_path(pid.cast_signed()) + let owners = find_socket_inode_owners(inode, entrypoint_pid)?; + Ok(TcpPeerSocketOwners { inode, owners }) +} + +/// Resolve exactly one owner for the TCP peer, failing closed on ambiguity. +#[cfg(target_os = "linux")] +fn resolve_single_tcp_peer_owner(entrypoint_pid: u32, peer_port: u16) -> Result { + let socket_owners = resolve_tcp_peer_socket_owners(entrypoint_pid, peer_port)?; + match socket_owners.owners.as_slice() { + [owner] => Ok(owner.clone()), + owners => { + let mut pids: Vec = owners.iter().map(|owner| owner.pid).collect(); + pids.sort_unstable(); + Err(miette::miette!( + "Ambiguous socket ownership for inode {}: PIDs [{}] all hold the same socket", + socket_owners.inode, + pids.iter() + .map(u32::to_string) + .collect::>() + .join(", ") + )) + } + } } /// Like `resolve_tcp_peer_binary`, but also returns the PID that owns the socket. /// -/// Needed for the ancestor walk: we must know the PID to walk `/proc//status` PPid chain. +/// Needed for the ancestor walk: we must know the PID to walk `/proc//status` `PPid` chain. #[cfg(target_os = "linux")] pub fn resolve_tcp_peer_identity(entrypoint_pid: u32, peer_port: u16) -> Result<(PathBuf, u32)> { - let inode = parse_proc_net_tcp(entrypoint_pid, peer_port)?; - let pid = find_pid_by_socket_inode(inode, entrypoint_pid)?; - let path = binary_path(pid.cast_signed())?; - Ok((path, pid)) + let owner = resolve_single_tcp_peer_owner(entrypoint_pid, peer_port)?; + let path = binary_path(owner.pid.cast_signed())?; + Ok((path, owner.pid)) } /// Read the `PPid` (parent PID) from `/proc//status`. @@ -221,42 +336,61 @@ fn parse_proc_net_tcp(pid: u32, peer_port: u16) -> Result { )) } -/// Scan process tree to find which PID owns a given socket inode. +/// Scan `/proc` to find every PID that owns a given socket inode. /// /// First scans descendants of `entrypoint_pid` (most likely owners), then falls /// back to scanning all of `/proc`. Requires `CAP_SYS_PTRACE` to read /// `/proc//fd/` for processes running as a different user. #[cfg(target_os = "linux")] -fn find_pid_by_socket_inode(inode: u64, entrypoint_pid: u32) -> Result { +fn find_socket_inode_owners(inode: u64, entrypoint_pid: u32) -> Result> { let target = format!("socket:[{inode}]"); + let mut owners = Vec::new(); + let mut checked = HashSet::new(); // First: scan descendants of the entrypoint process - let descendants = collect_descendant_pids(entrypoint_pid); - - for &pid in &descendants { - if let Some(found) = check_pid_fds(pid, &target) { - return Ok(found); + let descendants = collect_descendant_pids_with_depth(entrypoint_pid); + + for descendant in &descendants { + checked.insert(descendant.pid); + if check_pid_fds(descendant.pid, &target) { + owners.push(SocketOwner { + pid: descendant.pid, + source: SocketOwnerSource::Descendant { + depth: descendant.depth, + }, + }); } } // Fallback: scan all of /proc in case the process isn't in the tree if let Ok(proc_dir) = std::fs::read_dir("/proc") { + let mut proc_pids = Vec::new(); for entry in proc_dir.flatten() { let name = entry.file_name(); - let pid: u32 = match name.to_string_lossy().parse() { - Ok(p) => p, - Err(_) => continue, - }; - // Skip PIDs we already checked - if descendants.contains(&pid) { + if let Ok(pid) = name.to_string_lossy().parse::() { + proc_pids.push(pid); + } + } + proc_pids.sort_unstable(); + + for pid in proc_pids { + if checked.contains(&pid) { continue; } - if let Some(found) = check_pid_fds(pid, &target) { - return Ok(found); + checked.insert(pid); + if check_pid_fds(pid, &target) { + owners.push(SocketOwner { + pid, + source: SocketOwnerSource::ProcFallback, + }); } } } + if !owners.is_empty() { + return Ok(owners); + } + Err(miette::miette!( "No process found owning socket inode {} \ (scanned {} descendants of entrypoint PID {}). \ @@ -270,37 +404,58 @@ fn find_pid_by_socket_inode(inode: u64, entrypoint_pid: u32) -> Result { /// Check if a PID has an fd pointing to the given socket target string. #[cfg(target_os = "linux")] -fn check_pid_fds(pid: u32, target: &str) -> Option { +fn check_pid_fds(pid: u32, target: &str) -> bool { let fd_dir = format!("/proc/{pid}/fd"); - let fds = std::fs::read_dir(&fd_dir).ok()?; + let Some(fds) = std::fs::read_dir(&fd_dir).ok() else { + return false; + }; for fd_entry in fds.flatten() { if let Ok(link) = std::fs::read_link(fd_entry.path()) && link.to_string_lossy() == target { - return Some(pid); + return true; } } - None + false } /// Collect all descendant PIDs of a root process using `/proc//task//children`. /// /// Performs a BFS walk of the process tree. If `/proc//task//children` /// is not available (requires `CONFIG_PROC_CHILDREN`), returns only the root PID. -#[cfg(target_os = "linux")] +#[cfg(all(test, target_os = "linux"))] fn collect_descendant_pids(root_pid: u32) -> Vec { - let mut pids = vec![root_pid]; + collect_descendant_pids_with_depth(root_pid) + .into_iter() + .map(|descendant| descendant.pid) + .collect() +} + +/// Collect descendant PIDs with BFS depth, deduping children reported by multiple tasks. +#[cfg(target_os = "linux")] +fn collect_descendant_pids_with_depth(root_pid: u32) -> Vec { + let mut pids = vec![DescendantPid { + pid: root_pid, + depth: 0, + }]; + let mut seen = HashSet::from([root_pid]); let mut i = 0; while i < pids.len() { - let pid = pids[i]; + let pid = pids[i].pid; + let child_depth = pids[i].depth + 1; let task_dir = format!("/proc/{pid}/task"); if let Ok(tasks) = std::fs::read_dir(&task_dir) { for task_entry in tasks.flatten() { let children_path = task_entry.path().join("children"); if let Ok(children_str) = std::fs::read_to_string(&children_path) { for child in children_str.split_whitespace() { - if let Ok(child_pid) = child.parse::() { - pids.push(child_pid); + if let Ok(child_pid) = child.parse::() + && seen.insert(child_pid) + { + pids.push(DescendantPid { + pid: child_pid, + depth: child_depth, + }); } } } @@ -325,7 +480,7 @@ pub fn file_sha256(path: &Path) -> Result { let mut file = std::fs::File::open(path) .map_err(|e| miette::miette!("Failed to open {}: {e}", path.display()))?; let mut hasher = Sha256::new(); - let mut buf = [0u8; 65536]; + let mut buf = vec![0u8; 65536].into_boxed_slice(); let mut total_read = 0u64; loop { let n = file @@ -353,6 +508,52 @@ mod tests { use super::*; use std::io::Write; + /// Block until `/proc//exe` points at `target`. `Command::spawn` returns + /// once the child is scheduled, not once it has completed `exec()`; on + /// contended runners the readlink can still show the parent (test harness) + /// binary for a brief window. Byte-level `starts_with` tolerates the kernel's + /// `" (deleted)"` suffix on unlinked executables. + #[cfg(target_os = "linux")] + fn wait_for_child_exec(pid: i32, target: &Path) { + use std::os::unix::ffi::OsStrExt as _; + let target_bytes = target.as_os_str().as_bytes(); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(2); + loop { + if let Ok(link) = std::fs::read_link(format!("/proc/{pid}/exe")) + && link.as_os_str().as_bytes().starts_with(target_bytes) + { + return; + } + assert!( + std::time::Instant::now() < deadline, + "child pid {pid} did not exec into {target:?} within 2s" + ); + std::thread::sleep(std::time::Duration::from_millis(10)); + } + } + + /// Retry `Command::spawn` on `ETXTBSY`. The kernel rejects `execve` when + /// `inode->i_writecount > 0`, and the release of that counter after the + /// writer fd is closed isn't synchronous with `close(2)` under contention — + /// so the very-next-instruction `execve` can still race it. Any other error + /// surfaces immediately. + #[cfg(target_os = "linux")] + fn spawn_retrying_on_etxtbsy(cmd: &mut std::process::Command) -> std::process::Child { + let mut attempts = 0; + loop { + match cmd.spawn() { + Ok(child) => return child, + Err(err) + if err.kind() == std::io::ErrorKind::ExecutableFileBusy && attempts < 20 => + { + attempts += 1; + std::thread::sleep(std::time::Duration::from_millis(50)); + } + Err(err) => panic!("spawn failed after {attempts} ETXTBSY retries: {err}"), + } + } + } + #[test] fn file_sha256_computes_correct_hash() { let mut tmp = tempfile::NamedTempFile::new().unwrap(); @@ -391,6 +592,153 @@ mod tests { assert!(path.exists()); } + /// Verify that an unlinked binary's path is returned without the + /// kernel's " (deleted)" suffix. This is the common case during a + /// `docker cp` hot-swap of the supervisor binary — before this strip, + /// callers that `stat()` the returned path get `ENOENT` and the + /// ancestor integrity check in the CONNECT proxy denies every request. + #[cfg(target_os = "linux")] + #[test] + fn binary_path_strips_deleted_suffix() { + use std::os::unix::fs::PermissionsExt; + + // Copy /bin/sleep to a temp path we control so we can unlink it. + let tmp = tempfile::TempDir::new().unwrap(); + let exe_path = tmp.path().join("deleted-sleep"); + std::fs::copy("/bin/sleep", &exe_path).unwrap(); + std::fs::set_permissions(&exe_path, std::fs::Permissions::from_mode(0o755)).unwrap(); + + // Spawn a child from the temp binary, then unlink it while the + // child is still running. The child keeps the exec mapping via + // `/proc//exe`, but readlink will now return the tainted + // " (deleted)" string. + let mut cmd = std::process::Command::new(&exe_path); + cmd.arg("5"); + let mut child = spawn_retrying_on_etxtbsy(&mut cmd); + let pid: i32 = child.id().cast_signed(); + wait_for_child_exec(pid, &exe_path); + std::fs::remove_file(&exe_path).unwrap(); + + // Sanity check: the raw readlink should contain " (deleted)". + let raw = std::fs::read_link(format!("/proc/{pid}/exe")) + .unwrap() + .to_string_lossy() + .into_owned(); + assert!( + raw.ends_with(" (deleted)"), + "kernel should append ' (deleted)' to unlinked exe readlink; got {raw:?}" + ); + + // The public API should return the stripped path, not the tainted one. + let resolved = binary_path(pid).expect("binary_path should succeed for deleted binary"); + assert_eq!( + resolved, exe_path, + "binary_path should strip the ' (deleted)' suffix" + ); + let resolved_str = resolved.to_string_lossy(); + assert!( + !resolved_str.contains("(deleted)"), + "stripped path must not contain '(deleted)'; got {resolved_str:?}" + ); + + let _ = child.kill(); + let _ = child.wait(); + } + + /// A live executable whose basename literally ends with `" (deleted)"` + /// must be returned unchanged — we only strip when `stat()` reports + /// the raw readlink target missing. This guards against the trusted + /// identity source misattributing a running binary to a truncated + /// sibling path. + #[cfg(target_os = "linux")] + #[test] + fn binary_path_preserves_live_deleted_basename() { + use std::os::unix::fs::PermissionsExt; + + let tmp = tempfile::TempDir::new().unwrap(); + // Basename literally ends with " (deleted)" while the file is still + // on disk — a pathological but legal filename. + let exe_path = tmp.path().join("sleepy (deleted)"); + std::fs::copy("/bin/sleep", &exe_path).unwrap(); + std::fs::set_permissions(&exe_path, std::fs::Permissions::from_mode(0o755)).unwrap(); + + let mut cmd = std::process::Command::new(&exe_path); + cmd.arg("5"); + let mut child = spawn_retrying_on_etxtbsy(&mut cmd); + let pid: i32 = child.id().cast_signed(); + wait_for_child_exec(pid, &exe_path); + + // File is still linked — binary_path must return the path unchanged, + // suffix and all. + let resolved = binary_path(pid).expect("binary_path should succeed for live binary"); + assert_eq!( + resolved, exe_path, + "binary_path must NOT strip ' (deleted)' from a live executable's basename" + ); + assert!( + resolved.to_string_lossy().ends_with(" (deleted)"), + "stripped path unexpectedly trimmed a real filename: {resolved:?}" + ); + + let _ = child.kill(); + let _ = child.wait(); + } + + /// An unlinked executable whose filename contains non-UTF-8 bytes must + /// still strip exactly one kernel-added `" (deleted)"` suffix. We operate + /// on raw bytes via `OsStrExt`, so invalid UTF-8 is not a reason to skip + /// the strip and return a path that downstream `stat()` calls will reject. + #[cfg(target_os = "linux")] + #[test] + fn binary_path_strips_suffix_for_non_utf8_filename() { + use std::ffi::OsString; + use std::os::unix::ffi::{OsStrExt, OsStringExt}; + use std::os::unix::fs::PermissionsExt; + + let tmp = tempfile::TempDir::new().unwrap(); + // 0xFF is not valid UTF-8. Build the filename on raw bytes. + let mut raw_name: Vec = b"badname-".to_vec(); + raw_name.push(0xFF); + raw_name.extend_from_slice(b".bin"); + let exe_path = tmp.path().join(OsString::from_vec(raw_name)); + + std::fs::copy("/bin/sleep", &exe_path).unwrap(); + std::fs::set_permissions(&exe_path, std::fs::Permissions::from_mode(0o755)).unwrap(); + + let mut cmd = std::process::Command::new(&exe_path); + cmd.arg("5"); + let mut child = spawn_retrying_on_etxtbsy(&mut cmd); + let pid: i32 = child.id().cast_signed(); + wait_for_child_exec(pid, &exe_path); + std::fs::remove_file(&exe_path).unwrap(); + + // Sanity: raw readlink ends with " (deleted)" and is not valid UTF-8. + let raw = std::fs::read_link(format!("/proc/{pid}/exe")).unwrap(); + let raw_bytes = raw.as_os_str().as_bytes(); + assert!( + raw_bytes.ends_with(b" (deleted)"), + "kernel should append ' (deleted)' to unlinked exe readlink" + ); + assert!( + std::str::from_utf8(raw_bytes).is_err(), + "test precondition: raw readlink must contain non-UTF-8 bytes" + ); + + let resolved = + binary_path(pid).expect("binary_path should succeed for non-UTF-8 unlinked path"); + assert_eq!( + resolved, exe_path, + "binary_path must strip exactly one ' (deleted)' suffix for non-UTF-8 paths" + ); + assert!( + !resolved.as_os_str().as_bytes().ends_with(b" (deleted)"), + "stripped path must not end with ' (deleted)'" + ); + + let _ = child.kill(); + let _ = child.wait(); + } + #[cfg(target_os = "linux")] #[test] fn collect_descendants_includes_self() { @@ -399,6 +747,77 @@ mod tests { assert!(pids.contains(&pid)); } + #[cfg(target_os = "linux")] + #[test] + fn collect_descendants_dedupes_pids() { + let pid = std::process::id(); + let pids = collect_descendant_pids(pid); + let unique = pids.iter().copied().collect::>(); + assert_eq!(pids.len(), unique.len()); + } + + #[cfg(target_os = "linux")] + #[test] + fn resolve_tcp_peer_socket_owners_returns_all_forked_socket_holders() { + use std::net::{TcpListener, TcpStream}; + use std::time::{Duration, Instant}; + + let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); + let listener_port = listener.local_addr().unwrap().port(); + let stream = TcpStream::connect(("127.0.0.1", listener_port)).expect("connect"); + let peer_port = stream.local_addr().unwrap().port(); + let (_accepted, _) = listener.accept().expect("accept"); + + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + let child_pid = unsafe { libc::fork() }; + assert!(child_pid >= 0, "fork failed"); + if child_pid == 0 { + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + unsafe { + libc::sleep(30); + libc::_exit(0); + } + } + + let child_pid_u32 = child_pid.cast_unsigned(); + let entrypoint_pid = std::process::id(); + let deadline = Instant::now() + Duration::from_secs(2); + let owners = loop { + let owners = resolve_tcp_peer_socket_owners(entrypoint_pid, peer_port) + .expect("resolve socket owners"); + let owner_pids = owners + .owners + .iter() + .map(|owner| owner.pid) + .collect::>(); + if owner_pids.contains(&entrypoint_pid) && owner_pids.contains(&child_pid_u32) { + break owners; + } + assert!( + Instant::now() < deadline, + "timed out waiting for forked child to appear as a socket owner; got {owner_pids:?}" + ); + std::thread::sleep(Duration::from_millis(20)); + }; + + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + unsafe { + libc::kill(child_pid, libc::SIGKILL); + libc::waitpid(child_pid, std::ptr::null_mut(), 0); + } + + let owner_pids = owners + .owners + .iter() + .map(|owner| owner.pid) + .collect::>(); + assert!(owner_pids.contains(&entrypoint_pid)); + assert!(owner_pids.contains(&child_pid_u32)); + } + #[cfg(target_os = "linux")] #[test] #[allow(clippy::similar_names)] diff --git a/crates/openshell-sandbox/src/proxy.rs b/crates/openshell-sandbox/src/proxy.rs index 1b87a0c7f2..5344374ac3 100644 --- a/crates/openshell-sandbox/src/proxy.rs +++ b/crates/openshell-sandbox/src/proxy.rs @@ -6,7 +6,7 @@ use crate::denial_aggregator::DenialEvent; use crate::identity::BinaryIdentityCache; use crate::l7::tls::ProxyTlsState; -use crate::opa::{NetworkAction, OpaEngine}; +use crate::opa::{NetworkAction, OpaEngine, PolicyGenerationGuard}; use crate::policy::ProxyPolicy; use crate::secrets::{SecretResolver, rewrite_header_line}; use miette::{IntoDiagnostic, Result}; @@ -18,8 +18,10 @@ use openshell_ocsf::{ use std::net::{IpAddr, SocketAddr}; use std::path::PathBuf; use std::sync::Arc; -use std::sync::atomic::AtomicU32; -use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use std::sync::atomic::{AtomicU32, Ordering}; +use tokio::io::{ + AsyncRead as TokioAsyncRead, AsyncReadExt, AsyncWrite as TokioAsyncWrite, AsyncWriteExt, +}; use tokio::net::{TcpListener, TcpStream}; use tokio::sync::mpsc; use tokio::task::JoinHandle; @@ -27,6 +29,7 @@ use tracing::{debug, warn}; const MAX_HEADER_BYTES: usize = 8192; const INFERENCE_LOCAL_HOST: &str = "inference.local"; +const INFERENCE_LOCAL_PORT: u16 = 443; /// Maximum total bytes for a streaming inference response body (32 MiB). const MAX_STREAMING_BODY: usize = 32 * 1024 * 1024; @@ -41,6 +44,8 @@ const CHUNK_IDLE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(1 /// Result of a proxy CONNECT policy decision. struct ConnectDecision { action: NetworkAction, + /// Policy generation used for the L4 network decision. + generation: u64, /// Resolved binary path. binary: Option, /// PID owning the socket. @@ -78,6 +83,9 @@ pub struct InferenceContext { } impl InferenceContext { + // `router`/`routes` are intentionally distinct nouns (the router and the + // route list it consumes); both names are clearer than alternatives. + #[allow(clippy::similar_names)] pub fn new( patterns: Vec, router: openshell_router::Router, @@ -294,6 +302,10 @@ fn emit_denial_simple( } } +// Many distinct, non-related context parameters are required for a CONNECT +// dispatch; bundling them into a struct would just shift the noise into call +// sites. +#[allow(clippy::too_many_arguments)] async fn handle_tcp_connection( mut client: TcpStream, opa_engine: Arc, @@ -354,7 +366,7 @@ async fn handle_tcp_connection( let (host, port) = parse_target(target)?; let host_lc = host.to_ascii_lowercase(); - if host_lc == INFERENCE_LOCAL_HOST { + if host_lc == INFERENCE_LOCAL_HOST && port == INFERENCE_LOCAL_PORT { respond(&mut client, b"HTTP/1.1 200 Connection Established\r\n\r\n").await?; let outcome = handle_inference_interception( client, @@ -482,6 +494,8 @@ async fn handle_tcp_connection( return Ok(()); } + let sandbox_entrypoint_pid = entrypoint_pid.load(Ordering::Acquire); + // Query allowed_ips from the matched endpoint config (if any). // When present, the SSRF check validates resolved IPs against this // allowlist instead of blanket-blocking all private IPs. @@ -494,58 +508,67 @@ async fn handle_tcp_connection( // Defense-in-depth: resolve DNS and reject connections to internal IPs. let dns_connect_start = std::time::Instant::now(); + // The "non-empty" branch is the explicit-allowlist path; reading it first + // matches the policy decision narrative. + #[allow(clippy::if_not_else)] let mut upstream = if !raw_allowed_ips.is_empty() { // allowed_ips mode: validate resolved IPs against CIDR allowlist. // Loopback and link-local are still always blocked. match parse_allowed_ips(&raw_allowed_ips) { - Ok(nets) => match resolve_and_check_allowed_ips(&host, port, &nets).await { - Ok(addrs) => TcpStream::connect(addrs.as_slice()) + Ok(nets) => { + match resolve_and_check_allowed_ips(&host, port, &nets, sandbox_entrypoint_pid) .await - .into_diagnostic()?, - Err(reason) => { - { - let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) - .actor_process( - Process::from_bypass(&binary_str, &pid_str, &ancestors_str) - .with_cmd_line(&cmdline_str), - ) - .firewall_rule("-", "ssrf") - .message(format!( - "CONNECT blocked: allowed_ips check failed for {host_lc}:{port}" - )) - .status_detail(&reason) - .build(); - ocsf_emit!(event); + { + Ok(addrs) => TcpStream::connect(addrs.as_slice()) + .await + .into_diagnostic()?, + Err(reason) => { + { + let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Open) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .dst_endpoint(Endpoint::from_domain(&host_lc, port)) + .src_endpoint_addr(peer_addr.ip(), peer_addr.port()) + .actor_process( + Process::from_bypass(&binary_str, &pid_str, &ancestors_str) + .with_cmd_line(&cmdline_str), + ) + .firewall_rule("-", "ssrf") + .message(format!( + "CONNECT blocked: allowed_ips check failed for {host_lc}:{port}" + )) + .status_detail(&reason) + .build(); + ocsf_emit!(event); + } + emit_denial( + &denial_tx, + &host_lc, + port, + &binary_str, + &decision, + &reason, + "ssrf", + ); + respond( + &mut client, + &build_json_error_response( + 403, + "Forbidden", + "ssrf_denied", + &format!( + "CONNECT {host_lc}:{port} blocked: allowed_ips check failed" + ), + ), + ) + .await?; + return Ok(()); } - emit_denial( - &denial_tx, - &host_lc, - port, - &binary_str, - &decision, - &reason, - "ssrf", - ); - respond( - &mut client, - &build_json_error_response( - 403, - "Forbidden", - "ssrf_denied", - &format!("CONNECT {host_lc}:{port} blocked: allowed_ips check failed"), - ), - ) - .await?; - return Ok(()); } - }, + } Err(reason) => { { let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) @@ -592,7 +615,7 @@ async fn handle_tcp_connection( } } else { // Default: reject all internal IPs (loopback, RFC 1918, link-local). - match resolve_and_reject_internal(&host, port).await { + match resolve_and_reject_internal(&host, port, sandbox_entrypoint_pid).await { Ok(addrs) => TcpStream::connect(addrs.as_slice()) .await .into_diagnostic()?, @@ -649,12 +672,16 @@ async fn handle_tcp_connection( respond(&mut client, b"HTTP/1.1 200 Connection Established\r\n\r\n").await?; - // Check if endpoint has L7 config for protocol-aware inspection - let l7_config = query_l7_config(&opa_engine, &decision, &host_lc, port); + // Check if endpoint has L7 config for protocol-aware inspection, and + // retain the generation for HTTP passthrough keep-alive tunnels. + let l7_route = query_l7_route_snapshot(&opa_engine, &decision, &host_lc, port); // Log the allowed CONNECT — use CONNECT_L7 when L7 inspection follows, // so log consumers can distinguish L4-only decisions from tunnel lifecycle events. - let connect_msg = if l7_config.is_some() { + let connect_msg = if l7_route + .as_ref() + .is_some_and(|route| !route.configs.is_empty()) + { "CONNECT_L7" } else { "CONNECT" @@ -739,35 +766,56 @@ async fn handle_tcp_connection( crate::l7::tls::tls_connect_upstream(upstream, &host_lc, tls.upstream_config()) .await?; - if let Some(ref l7_config) = l7_config { + if let Some(route) = l7_route.as_ref().filter(|route| !route.configs.is_empty()) { // L7 inspection on terminated TLS traffic. - let tunnel_engine = opa_engine.clone_engine_for_tunnel().unwrap_or_else(|e| { - let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Low) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .message(format!( - "Failed to clone OPA engine for L7, falling back to relay-only: {e}" - )) - .build(); - ocsf_emit!(event); - regorus::Engine::new() - }); - crate::l7::relay::relay_with_inspection( - l7_config, - std::sync::Mutex::new(tunnel_engine), - &mut tls_client, - &mut tls_upstream, - &ctx, - ) - .await + let tunnel_engine = match opa_engine.clone_engine_for_tunnel(route.generation) { + Ok(engine) => engine, + Err(e) => { + emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); + return Ok(()); + } + }; + if route.configs.len() == 1 { + crate::l7::relay::relay_with_inspection( + &route.configs[0].config, + tunnel_engine, + &mut tls_client, + &mut tls_upstream, + &ctx, + ) + .await + } else { + let configs: Vec = route + .configs + .iter() + .map(|snapshot| snapshot.config.clone()) + .collect(); + crate::l7::relay::relay_with_route_selection( + &configs, + tunnel_engine, + &mut tls_client, + &mut tls_upstream, + &ctx, + ) + .await + } } else { // No L7 config — relay with credential injection only. + let generation = l7_route + .as_ref() + .map_or(decision.generation, |route| route.generation); + let generation_guard = match opa_engine.generation_guard(generation) { + Ok(guard) => guard, + Err(e) => { + emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); + return Ok(()); + } + }; crate::l7::relay::relay_passthrough_with_credentials( &mut tls_client, &mut tls_upstream, &ctx, + &generation_guard, ) .await } @@ -810,29 +858,39 @@ async fn handle_tcp_connection( } } else if is_http { // Plaintext HTTP detected. - if let Some(ref l7_config) = l7_config { - let tunnel_engine = opa_engine.clone_engine_for_tunnel().unwrap_or_else(|e| { - let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Low) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .message(format!( - "Failed to clone OPA engine for L7, falling back to relay-only: {e}" - )) - .build(); - ocsf_emit!(event); - regorus::Engine::new() - }); - if let Err(e) = crate::l7::relay::relay_with_inspection( - l7_config, - std::sync::Mutex::new(tunnel_engine), - &mut client, - &mut upstream, - &ctx, - ) - .await - { + if let Some(route) = l7_route.as_ref().filter(|route| !route.configs.is_empty()) { + let tunnel_engine = match opa_engine.clone_engine_for_tunnel(route.generation) { + Ok(engine) => engine, + Err(e) => { + emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); + return Ok(()); + } + }; + let relay_result = if route.configs.len() == 1 { + crate::l7::relay::relay_with_inspection( + &route.configs[0].config, + tunnel_engine, + &mut client, + &mut upstream, + &ctx, + ) + .await + } else { + let configs: Vec = route + .configs + .iter() + .map(|snapshot| snapshot.config.clone()) + .collect(); + crate::l7::relay::relay_with_route_selection( + &configs, + tunnel_engine, + &mut client, + &mut upstream, + &ctx, + ) + .await + }; + if let Err(e) = relay_result { if is_benign_relay_error(&e) { debug!(host = %host_lc, port = port, error = %e, "L7 connection closed"); } else { @@ -848,10 +906,21 @@ async fn handle_tcp_connection( } } else { // Plaintext HTTP, no L7 config — relay with credential injection. + let generation = l7_route + .as_ref() + .map_or(decision.generation, |route| route.generation); + let generation_guard = match opa_engine.generation_guard(generation) { + Ok(guard) => guard, + Err(e) => { + emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); + return Ok(()); + } + }; if let Err(e) = crate::l7::relay::relay_passthrough_with_credentials( &mut client, &mut upstream, &ctx, + &generation_guard, ) .await { @@ -884,6 +953,185 @@ async fn handle_tcp_connection( Ok(()) } +/// Resolved process identity for a TCP peer: binary path, PID, ancestor chain, +/// cmdline paths, and the TOFU-verified binary hash. +/// +/// Produced by [`resolve_process_identity`]; consumed by [`evaluate_opa_tcp`] +/// and by the identity-chain regression tests. +#[cfg(target_os = "linux")] +struct ResolvedIdentity { + bin_path: PathBuf, + binary_pid: u32, + ancestors: Vec, + cmdline_paths: Vec, + bin_hash: String, +} + +#[cfg(target_os = "linux")] +#[derive(Debug, Eq, PartialEq)] +struct PolicyIdentityKey { + bin_path: PathBuf, + ancestors: Vec, + cmdline_paths: Vec, + bin_hash: String, +} + +#[cfg(target_os = "linux")] +impl ResolvedIdentity { + fn policy_key(&self) -> PolicyIdentityKey { + PolicyIdentityKey { + bin_path: self.bin_path.clone(), + ancestors: self.ancestors.clone(), + cmdline_paths: self.cmdline_paths.clone(), + bin_hash: self.bin_hash.clone(), + } + } +} + +/// Error from [`resolve_process_identity`]. Carries the deny reason and +/// whatever partial identity data was resolved before the failure so the +/// caller can include it in the [`ConnectDecision`] and OCSF event. +#[cfg(target_os = "linux")] +struct IdentityError { + reason: String, + binary: Option, + binary_pid: Option, + ancestors: Vec, +} + +#[cfg(target_os = "linux")] +fn resolve_owner_identity( + owner_pid: u32, + entrypoint_pid: u32, + identity_cache: &BinaryIdentityCache, +) -> std::result::Result { + let bin_path = + crate::procfs::binary_path(owner_pid.cast_signed()).map_err(|e| IdentityError { + reason: format!("failed to resolve peer binary for PID {owner_pid}: {e}"), + binary: None, + binary_pid: Some(owner_pid), + ancestors: vec![], + })?; + + let bin_hash = identity_cache + .verify_or_cache(&bin_path) + .map_err(|e| IdentityError { + reason: format!("binary integrity check failed: {e}"), + binary: Some(bin_path.clone()), + binary_pid: Some(owner_pid), + ancestors: vec![], + })?; + + let ancestors = crate::procfs::collect_ancestor_binaries(owner_pid, entrypoint_pid); + + for ancestor in &ancestors { + identity_cache + .verify_or_cache(ancestor) + .map_err(|e| IdentityError { + reason: format!( + "ancestor integrity check failed for {}: {e}", + ancestor.display() + ), + binary: Some(bin_path.clone()), + binary_pid: Some(owner_pid), + ancestors: ancestors.clone(), + })?; + } + + let mut exclude = ancestors.clone(); + exclude.push(bin_path.clone()); + let cmdline_paths = crate::procfs::collect_cmdline_paths(owner_pid, entrypoint_pid, &exclude); + + Ok(ResolvedIdentity { + bin_path, + binary_pid: owner_pid, + ancestors, + cmdline_paths, + bin_hash, + }) +} + +/// Resolve the identity of the process owning a TCP peer connection. +/// +/// Walks `/proc//net/tcp` to find the socket inode, locates +/// every owning PID, reads `/proc//exe`, TOFU-verifies each binary hash, +/// walks each ancestor chain verifying every ancestor, and collects +/// cmdline-derived absolute paths for script detection. +/// +/// This is the identity-resolution block of [`evaluate_opa_tcp`] extracted +/// into a standalone helper so it can be exercised by Linux-only regression +/// tests without a full OPA engine. The key invariant under test is that on +/// a hot-swap of the peer binary, the failure mode is +/// `"Binary integrity violation"` (from the identity cache) rather than +/// `"Failed to stat ... (deleted)"` (from the kernel-tainted path). +#[cfg(target_os = "linux")] +fn resolve_process_identity( + entrypoint_pid: u32, + peer_port: u16, + identity_cache: &BinaryIdentityCache, +) -> std::result::Result { + let socket_owners = crate::procfs::resolve_tcp_peer_socket_owners(entrypoint_pid, peer_port) + .map_err(|e| IdentityError { + reason: format!("failed to resolve peer binary: {e}"), + binary: None, + binary_pid: None, + ancestors: vec![], + })?; + + let mut identities = Vec::with_capacity(socket_owners.owners.len()); + for owner in &socket_owners.owners { + identities.push(resolve_owner_identity( + owner.pid, + entrypoint_pid, + identity_cache, + )?); + } + + let Some(first_identity) = identities.first() else { + return Err(IdentityError { + reason: format!( + "failed to resolve peer binary: no process found owning socket inode {}", + socket_owners.inode + ), + binary: None, + binary_pid: None, + ancestors: vec![], + }); + }; + + let first_key = first_identity.policy_key(); + if identities + .iter() + .skip(1) + .any(|identity| identity.policy_key() != first_key) + { + let mut pids: Vec = identities + .iter() + .map(|identity| identity.binary_pid) + .collect(); + pids.sort_unstable(); + return Err(IdentityError { + reason: format!( + "ambiguous shared socket ownership: inode {} is held by PIDs [{}] with different policy identities", + socket_owners.inode, + pids.iter() + .map(u32::to_string) + .collect::>() + .join(", ") + ), + binary: None, + binary_pid: None, + ancestors: vec![], + }); + } + + let mut identity = identities.swap_remove(0); + if let Some(lowest_pid) = socket_owners.owners.iter().map(|owner| owner.pid).min() { + identity.binary_pid = lowest_pid; + } + Ok(identity) +} + /// Evaluate OPA policy for a TCP connection with identity binding via /proc/net/tcp. #[cfg(target_os = "linux")] fn evaluate_opa_tcp( @@ -905,6 +1153,7 @@ fn evaluate_opa_tcp( -> ConnectDecision { ConnectDecision { action: NetworkAction::Deny { reason }, + generation: engine.current_generation(), binary, binary_pid, ancestors, @@ -926,55 +1175,26 @@ fn evaluate_opa_tcp( let total_start = std::time::Instant::now(); let peer_port = peer_addr.port(); - let (bin_path, binary_pid) = match crate::procfs::resolve_tcp_peer_identity(pid, peer_port) { - Ok(r) => r, - Err(e) => { - return deny( - format!("failed to resolve peer binary: {e}"), - None, - None, - vec![], - vec![], - ); - } - }; - - // TOFU verify the immediate binary - let bin_hash = match identity_cache.verify_or_cache(&bin_path) { - Ok(h) => h, - Err(e) => { + let identity = match resolve_process_identity(pid, peer_port, identity_cache) { + Ok(id) => id, + Err(err) => { return deny( - format!("binary integrity check failed: {e}"), - Some(bin_path), - Some(binary_pid), - vec![], + err.reason, + err.binary, + err.binary_pid, + err.ancestors, vec![], ); } }; - // Walk the process tree upward to collect ancestor binaries - let ancestors = crate::procfs::collect_ancestor_binaries(binary_pid, pid); - - for ancestor in &ancestors { - if let Err(e) = identity_cache.verify_or_cache(ancestor) { - return deny( - format!( - "ancestor integrity check failed for {}: {e}", - ancestor.display() - ), - Some(bin_path), - Some(binary_pid), - ancestors.clone(), - vec![], - ); - } - } - - // Collect cmdline paths for script-based binary detection. - let mut exclude = ancestors.clone(); - exclude.push(bin_path.clone()); - let cmdline_paths = crate::procfs::collect_cmdline_paths(binary_pid, pid, &exclude); + let ResolvedIdentity { + bin_path, + binary_pid, + ancestors, + cmdline_paths, + bin_hash, + } = identity; let input = NetworkInput { host: host.to_string(), @@ -985,9 +1205,10 @@ fn evaluate_opa_tcp( cmdline_paths: cmdline_paths.clone(), }; - let result = match engine.evaluate_network_action(&input) { - Ok(action) => ConnectDecision { + let result = match engine.evaluate_network_action_with_generation(&input) { + Ok((action, generation)) => ConnectDecision { action, + generation, binary: Some(bin_path), binary_pid: Some(binary_pid), ancestors, @@ -1012,7 +1233,7 @@ fn evaluate_opa_tcp( #[cfg(not(target_os = "linux"))] fn evaluate_opa_tcp( _peer_addr: SocketAddr, - _engine: &OpaEngine, + engine: &OpaEngine, _identity_cache: &BinaryIdentityCache, _entrypoint_pid: &AtomicU32, _host: &str, @@ -1022,6 +1243,7 @@ fn evaluate_opa_tcp( action: NetworkAction::Deny { reason: "identity binding unavailable on this platform".into(), }, + generation: engine.current_generation(), binary: None, binary_pid: None, ancestors: vec![], @@ -1205,9 +1427,6 @@ async fn route_inference_request( ocsf_emit!(event); } - // Strip credential + framing/hop-by-hop headers. - let filtered_headers = sanitize_inference_request_headers(&request.headers); - let routes = ctx.routes.read().await; if routes.is_empty() { @@ -1231,7 +1450,7 @@ async fn route_inference_request( &pattern.protocol, &request.method, &normalized_path, - filtered_headers, + request.headers.clone(), bytes::Bytes::from(request.body.clone()), &routes, ) @@ -1393,14 +1612,6 @@ fn router_error_to_http(err: &openshell_router::RouterError) -> (u16, String) { } } -fn sanitize_inference_request_headers(headers: &[(String, String)]) -> Vec<(String, String)> { - headers - .iter() - .filter(|(name, _)| !should_strip_request_header(name)) - .cloned() - .collect() -} - fn sanitize_inference_response_headers(headers: Vec<(String, String)>) -> Vec<(String, String)> { headers .into_iter() @@ -1408,14 +1619,6 @@ fn sanitize_inference_response_headers(headers: Vec<(String, String)>) -> Vec<(S .collect() } -fn should_strip_request_header(name: &str) -> bool { - let name_lc = name.to_ascii_lowercase(); - matches!( - name_lc.as_str(), - "authorization" | "x-api-key" | "host" | "content-length" - ) || is_hop_by_hop_header(&name_lc) -} - fn should_strip_response_header(name: &str) -> bool { let name_lc = name.to_ascii_lowercase(); matches!(name_lc.as_str(), "content-length") || is_hop_by_hop_header(&name_lc) @@ -1444,20 +1647,46 @@ async fn write_all(writer: &mut (impl tokio::io::AsyncWrite + Unpin), data: &[u8 Ok(()) } +#[derive(Debug, Clone)] +struct L7ConfigSnapshot { + config: crate::l7::L7EndpointConfig, +} + +#[derive(Debug, Clone)] +struct L7RouteSnapshot { + configs: Vec, + generation: u64, +} + +fn emit_l7_tunnel_close_after_policy_change(host: &str, port: u16, error: miette::Report) { + let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Open) + .action(ActionId::Denied) + .disposition(DispositionId::Blocked) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .dst_endpoint(Endpoint::from_domain(host, port)) + .message(format!( + "L7 tunnel closed before inspection because policy changed: {error}" + )) + .build(); + ocsf_emit!(event); +} + /// Query L7 endpoint config from the OPA engine for a matched CONNECT decision. /// /// Returns `Some(L7EndpointConfig)` if the matched endpoint has L7 config (protocol field), /// `None` for L4-only endpoints. -fn query_l7_config( +fn query_l7_route_snapshot( engine: &OpaEngine, decision: &ConnectDecision, host: &str, port: u16, -) -> Option { +) -> Option { // Only query if action is Allow (not Deny) let has_policy = match &decision.action { NetworkAction::Allow { matched_policy } => matched_policy.is_some(), - _ => false, + NetworkAction::Deny { .. } => false, }; if !has_policy { return None; @@ -1472,9 +1701,15 @@ fn query_l7_config( cmdline_paths: decision.cmdline_paths.clone(), }; - match engine.query_endpoint_config(&input) { - Ok(Some(val)) => crate::l7::parse_l7_config(&val), - Ok(None) => None, + match engine.query_endpoint_configs_with_generation(&input) { + Ok((vals, generation)) => Some(L7RouteSnapshot { + configs: vals + .into_iter() + .filter_map(|val| crate::l7::parse_l7_config(&val)) + .map(|config| L7ConfigSnapshot { config }) + .collect(), + generation, + }), Err(e) => { let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) .activity(ActivityId::Fail) @@ -1489,6 +1724,16 @@ fn query_l7_config( } } +fn select_l7_config_for_path<'a>( + configs: &'a [L7ConfigSnapshot], + path: &str, +) -> Option<&'a L7ConfigSnapshot> { + configs + .iter() + .filter(|snapshot| snapshot.config.matches_path(path)) + .max_by_key(|snapshot| snapshot.config.path_specificity()) +} + /// Query the TLS mode for an endpoint, independent of L7 config. /// /// This extracts `tls: skip` from the endpoint even when no `protocol` is set. @@ -1500,7 +1745,7 @@ fn query_tls_mode( ) -> crate::l7::TlsMode { let has_policy = match &decision.action { NetworkAction::Allow { matched_policy } => matched_policy.is_some(), - _ => false, + NetworkAction::Deny { .. } => false, }; if !has_policy { return crate::l7::TlsMode::Auto; @@ -1530,7 +1775,8 @@ fn query_tls_mode( /// — synthesizing an `allowed_ips` entry for them would be silently /// un-enforceable at runtime. fn implicit_allowed_ips_for_ip_host(host: &str) -> Vec { - if let Ok(ip) = host.parse::() { + let lookup_host = normalize_host_lookup_key(host); + if let Ok(ip) = lookup_host.parse::() { if is_always_blocked_ip(ip) { warn!( host, @@ -1540,33 +1786,141 @@ fn implicit_allowed_ips_for_ip_host(host: &str) -> Vec { ); return vec![]; } - vec![host.to_string()] + vec![lookup_host.to_string()] } else { vec![] } } -/// Resolve DNS for a host:port and reject if any resolved address is internal. -/// -/// Returns the resolved `SocketAddr` list on success. Returns an error string -/// if any resolved IP is in an internal range or if DNS resolution fails. -async fn resolve_and_reject_internal( +fn normalize_host_lookup_key(host: &str) -> &str { + host.strip_prefix('[') + .and_then(|trimmed| trimmed.strip_suffix(']')) + .unwrap_or(host) +} + +fn resolve_ip_literal(host: &str, port: u16) -> Option> { + normalize_host_lookup_key(host) + .parse::() + .ok() + .map(|ip| vec![SocketAddr::new(ip, port)]) +} + +#[cfg(any(target_os = "linux", test))] +fn parse_hosts_file_for_host(contents: &str, host: &str) -> Vec { + let lookup_host = normalize_host_lookup_key(host); + let mut addrs = Vec::new(); + + for raw_line in contents.lines() { + let line = raw_line.split('#').next().unwrap_or("").trim(); + if line.is_empty() { + continue; + } + + let mut fields = line.split_whitespace(); + let Some(ip_str) = fields.next() else { + continue; + }; + let Ok(ip) = ip_str.parse::() else { + continue; + }; + + if fields.any(|alias| alias.eq_ignore_ascii_case(lookup_host)) && !addrs.contains(&ip) { + addrs.push(ip); + } + } + + addrs +} + +#[cfg(any(target_os = "linux", test))] +fn resolve_from_hosts_file_contents(contents: &str, host: &str, port: u16) -> Vec { + parse_hosts_file_for_host(contents, host) + .into_iter() + .map(|ip| SocketAddr::new(ip, port)) + .collect() +} + +#[cfg(target_os = "linux")] +async fn resolve_from_sandbox_hosts( + host: &str, + port: u16, + entrypoint_pid: u32, +) -> Option> { + if entrypoint_pid == 0 { + return None; + } + + let hosts_path = format!("/proc/{entrypoint_pid}/root/etc/hosts"); + let contents = match tokio::fs::read_to_string(&hosts_path).await { + Ok(contents) => contents, + Err(error) => { + debug!( + pid = entrypoint_pid, + path = %hosts_path, + host, + "Falling back to DNS; failed to read sandbox hosts file: {error}" + ); + return None; + } + }; + + let addrs = resolve_from_hosts_file_contents(&contents, host, port); + if addrs.is_empty() { None } else { Some(addrs) } +} + +// Mirrors the Linux signature so call sites can `.await` uniformly across +// platforms; the non-Linux path has nothing to await. +#[cfg(not(target_os = "linux"))] +#[allow(clippy::unused_async)] +async fn resolve_from_sandbox_hosts( + _host: &str, + _port: u16, + _entrypoint_pid: u32, +) -> Option> { + None +} + +async fn resolve_socket_addrs( host: &str, port: u16, + entrypoint_pid: u32, ) -> std::result::Result, String> { - let addrs: Vec = tokio::net::lookup_host((host, port)) + if let Some(addrs) = resolve_ip_literal(host, port) { + return Ok(addrs); + } + + if let Some(addrs) = resolve_from_sandbox_hosts(host, port, entrypoint_pid).await { + return Ok(addrs); + } + + let lookup_host = normalize_host_lookup_key(host); + let addrs: Vec = tokio::net::lookup_host((lookup_host, port)) .await - .map_err(|e| format!("DNS resolution failed for {host}:{port}: {e}"))? + .map_err(|e| format!("DNS resolution failed for {lookup_host}:{port}: {e}"))? .collect(); if addrs.is_empty() { return Err(format!( - "DNS resolution returned no addresses for {host}:{port}" + "DNS resolution returned no addresses for {lookup_host}:{port}" )); } - for addr in &addrs { - if is_internal_ip(addr.ip()) { + Ok(addrs) +} + +fn reject_internal_resolved_addrs( + host: &str, + addrs: &[SocketAddr], +) -> std::result::Result<(), String> { + if addrs.is_empty() { + return Err(format!( + "DNS resolution returned no addresses for {}", + normalize_host_lookup_key(host) + )); + } + + for addr in addrs { + if is_internal_ip(addr.ip()) { return Err(format!( "{host} resolves to internal address {}, connection rejected", addr.ip() @@ -1574,29 +1928,19 @@ async fn resolve_and_reject_internal( } } - Ok(addrs) + Ok(()) } -/// Resolve DNS and validate resolved addresses against a CIDR/IP allowlist. -/// -/// Rejects loopback and link-local unconditionally. For all other resolved -/// addresses, checks that each one matches at least one entry in `allowed_ips`. -/// Entries can be CIDR notation ("10.0.5.0/24") or exact IPs ("10.0.5.20"). -/// -/// Returns the resolved `SocketAddr` list on success. -async fn resolve_and_check_allowed_ips( +fn validate_allowed_ips_for_resolved_addrs( host: &str, port: u16, + addrs: &[SocketAddr], allowed_ips: &[ipnet::IpNet], -) -> std::result::Result, String> { - let addrs: Vec = tokio::net::lookup_host((host, port)) - .await - .map_err(|e| format!("DNS resolution failed for {host}:{port}: {e}"))? - .collect(); - +) -> std::result::Result<(), String> { if addrs.is_empty() { return Err(format!( - "DNS resolution returned no addresses for {host}:{port}" + "DNS resolution returned no addresses for {}", + normalize_host_lookup_key(host) )); } @@ -1607,7 +1951,7 @@ async fn resolve_and_check_allowed_ips( )); } - for addr in &addrs { + for addr in addrs { // Always block loopback and link-local if is_always_blocked_ip(addr.ip()) { return Err(format!( @@ -1626,6 +1970,40 @@ async fn resolve_and_check_allowed_ips( } } + Ok(()) +} + +/// Resolve a host:port using sandbox `/etc/hosts` first (when available), then +/// reject if any resolved address is internal. +/// +/// Returns the resolved `SocketAddr` list on success. Returns an error string +/// if any resolved IP is in an internal range or if DNS resolution fails. +async fn resolve_and_reject_internal( + host: &str, + port: u16, + entrypoint_pid: u32, +) -> std::result::Result, String> { + let addrs = resolve_socket_addrs(host, port, entrypoint_pid).await?; + reject_internal_resolved_addrs(host, &addrs)?; + Ok(addrs) +} + +/// Resolve a host:port using sandbox `/etc/hosts` first (when available), then +/// validate resolved addresses against a CIDR/IP allowlist. +/// +/// Rejects loopback and link-local unconditionally. For all other resolved +/// addresses, checks that each one matches at least one entry in `allowed_ips`. +/// Entries can be CIDR notation ("10.0.5.0/24") or exact IPs ("10.0.5.20"). +/// +/// Returns the resolved `SocketAddr` list on success. +async fn resolve_and_check_allowed_ips( + host: &str, + port: u16, + allowed_ips: &[ipnet::IpNet], + entrypoint_pid: u32, +) -> std::result::Result, String> { + let addrs = resolve_socket_addrs(host, port, entrypoint_pid).await?; + validate_allowed_ips_for_resolved_addrs(host, port, &addrs, allowed_ips)?; Ok(addrs) } @@ -1699,7 +2077,7 @@ fn parse_allowed_ips(raw: &[String]) -> std::result::Result, S } nets.push(n); } - Err(_) => errors.push(format!("invalid CIDR/IP in allowed_ips: {entry}")), + Err(()) => errors.push(format!("invalid CIDR/IP in allowed_ips: {entry}")), } } @@ -1710,7 +2088,7 @@ fn parse_allowed_ips(raw: &[String]) -> std::result::Result, S } } -/// Query allowed_ips from the matched endpoint config for a CONNECT decision. +/// Query `allowed_ips` from the matched endpoint config for a CONNECT decision. fn query_allowed_ips( engine: &OpaEngine, decision: &ConnectDecision, @@ -1720,7 +2098,7 @@ fn query_allowed_ips( // Only query if action is Allow with a matched policy let has_policy = match &decision.action { NetworkAction::Allow { matched_policy } => matched_policy.is_some(), - _ => false, + NetworkAction::Deny { .. } => false, }; if !has_policy { return vec![]; @@ -1753,15 +2131,21 @@ fn query_allowed_ips( } } +/// Canonicalize the request-target for inference pattern detection. +/// +/// Falls back to the raw path on canonicalization error: the request is then +/// routed through the normal forward path, where `rest.rs::parse_http_request` +/// will reject it properly. Returning the raw path here prevents a crafted +/// target from bypassing inference routing without our detection logic having +/// to implement a second, duplicate error-response surface. fn normalize_inference_path(path: &str) -> String { - if let Some(scheme_idx) = path.find("://") { - let after_scheme = &path[scheme_idx + 3..]; - if let Some(path_start) = after_scheme.find('/') { - return after_scheme[path_start..].to_string(); - } - return "/".to_string(); + match crate::l7::path::canonicalize_request_target( + path, + &crate::l7::path::CanonicalizeOptions::default(), + ) { + Ok((canon, _)) => canon.path, + Err(_) => path.to_string(), } - path.to_string() } /// Extract the hostname from an absolute-form URI used in plain HTTP proxy requests. @@ -1773,15 +2157,12 @@ fn normalize_inference_path(path: &str) -> String { fn extract_host_from_uri(uri: &str) -> String { // Absolute-form URIs look like "http://host[:port]/path" // Strip the scheme prefix, then extract the authority (host[:port]) before the first '/'. - let after_scheme = uri.find("://").map(|i| &uri[i + 3..]).unwrap_or(uri); + let after_scheme = uri.find("://").map_or(uri, |i| &uri[i + 3..]); let authority = after_scheme.split('/').next().unwrap_or(after_scheme); // Strip port if present (handle IPv6 bracket notation) let host = if authority.starts_with('[') { // IPv6: [::1]:port - authority - .find(']') - .map(|i| &authority[..=i]) - .unwrap_or(authority) + authority.find(']').map_or(authority, |i| &authority[..=i]) } else { authority.split(':').next().unwrap_or(authority) }; @@ -1816,14 +2197,12 @@ fn parse_proxy_uri(uri: &str) -> Result<(String, String, u16, String)> { .find(']') .ok_or_else(|| miette::miette!("Unclosed IPv6 bracket in URI: {uri}"))?; let after_bracket = &rest[bracket_end + 1..]; - if let Some(slash_pos) = after_bracket.find('/') { + after_bracket.find('/').map_or((rest, "/"), |slash_pos| { ( - &rest[..bracket_end + 1 + slash_pos], + &rest[..=bracket_end + slash_pos], &after_bracket[slash_pos..], ) - } else { - (&rest[..], "/") - } + }) } else if let Some(slash_pos) = rest.find('/') { (&rest[..slash_pos], &rest[slash_pos..]) } else { @@ -1934,10 +2313,10 @@ fn rewrite_forward_request( continue; } - let rewritten_line = match secret_resolver { - Some(resolver) => rewrite_header_line(line, resolver), - None => line.to_string(), - }; + let rewritten_line = secret_resolver.map_or_else( + || line.to_string(), + |resolver| rewrite_header_line(line, resolver), + ); output.extend_from_slice(rewritten_line.as_bytes()); output.extend_from_slice(b"\r\n"); @@ -1974,12 +2353,52 @@ fn rewrite_forward_request( Ok(output) } +async fn relay_rewritten_forward_request( + method: &str, + path: &str, + rewritten: Vec, + client: &mut C, + upstream: &mut U, + generation_guard: &PolicyGenerationGuard, +) -> Result +where + C: TokioAsyncRead + TokioAsyncWrite + Unpin, + U: TokioAsyncRead + TokioAsyncWrite + Unpin, +{ + let header_end = rewritten + .windows(4) + .position(|w| w == b"\r\n\r\n") + .map_or(rewritten.len(), |p| p + 4); + let header_str = String::from_utf8_lossy(&rewritten[..header_end]); + let body_length = crate::l7::rest::parse_body_length(&header_str)?; + let (_, query_params) = crate::l7::rest::parse_target_query(path)?; + let req = crate::l7::provider::L7Request { + action: method.to_string(), + target: path.to_string(), + query_params, + raw_header: rewritten, + body_length, + }; + + crate::l7::rest::relay_http_request_with_resolver_guarded( + &req, + client, + upstream, + None, + Some(generation_guard), + ) + .await +} + /// Handle a plain HTTP forward proxy request (non-CONNECT). /// /// Public IPs are allowed through when the endpoint passes OPA evaluation. /// Private IPs require explicit `allowed_ips` on the endpoint config (SSRF /// override). Rewrites the absolute-form request to origin-form, connects -/// upstream, and relays the response using `copy_bidirectional` for streaming. +/// upstream, and relays the request/response using the guarded HTTP relay. +// Many distinct, non-related context parameters are required for forward proxy +// dispatch; bundling them into a struct would just shift the noise into call sites. +#[allow(clippy::too_many_arguments)] async fn handle_forward_proxy( method: &str, target_uri: &str, @@ -1992,8 +2411,11 @@ async fn handle_forward_proxy( secret_resolver: Option>, denial_tx: Option<&mpsc::UnboundedSender>, ) -> Result<()> { - // 1. Parse the absolute-form URI - let (scheme, host, port, path) = match parse_proxy_uri(target_uri) { + // 1. Parse the absolute-form URI. `path` is marked `mut` so that, when an + // L7 config applies, the canonicalized form produced below replaces it + // in-place — keeping OPA evaluation and the bytes written onto the wire + // in sync. See the L7 block below. + let (scheme, host, port, mut path) = match parse_proxy_uri(target_uri) { Ok(parsed) => parsed, Err(e) => { let event = HttpActivityBuilder::new(crate::ocsf_ctx()) @@ -2132,23 +2554,72 @@ async fn handle_forward_proxy( } }; let policy_str = matched_policy.as_deref().unwrap_or("-"); + let sandbox_entrypoint_pid = entrypoint_pid.load(Ordering::Acquire); + let forward_generation_guard = match opa_engine.generation_guard(decision.generation) { + Ok(guard) => guard, + Err(e) => { + emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); + respond( + client, + &build_json_error_response( + 403, + "Forbidden", + "policy_denied", + &format!("{method} {host_lc}:{port}{path} not permitted by policy"), + ), + ) + .await?; + return Ok(()); + } + }; + let mut forward_request_bytes = buf[..used].to_vec(); + let mut upstream_target = path.clone(); // 4b. If the endpoint has L7 config, evaluate the request against // L7 policy. The forward proxy handles exactly one request per // connection (Connection: close), so a single evaluation suffices. - if let Some(l7_config) = query_l7_config(&opa_engine, &decision, &host_lc, port) { - let tunnel_engine = opa_engine.clone_engine_for_tunnel().unwrap_or_else(|e| { - let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Low) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .message(format!("Failed to clone OPA engine for forward L7: {e}")) - .build(); - ocsf_emit!(event); - regorus::Engine::new() - }); - let engine_mutex = std::sync::Mutex::new(tunnel_engine); + if let Some(route) = query_l7_route_snapshot(&opa_engine, &decision, &host_lc, port) + && !route.configs.is_empty() + { + if route.generation != forward_generation_guard.captured_generation() { + emit_l7_tunnel_close_after_policy_change( + &host_lc, + port, + miette::miette!( + "policy changed before forward L7 evaluation [expected_generation:{} current_generation:{}]", + forward_generation_guard.captured_generation(), + route.generation, + ), + ); + respond( + client, + &build_json_error_response( + 403, + "Forbidden", + "policy_denied", + &format!("{method} {host_lc}:{port}{path} not permitted by policy"), + ), + ) + .await?; + return Ok(()); + } + let tunnel_engine = match opa_engine.clone_engine_for_tunnel(route.generation) { + Ok(engine) => engine, + Err(e) => { + emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); + respond( + client, + &build_json_error_response( + 403, + "Forbidden", + "policy_denied", + &format!("{method} {host_lc}:{port}{path} not permitted by policy"), + ), + ) + .await?; + return Ok(()); + } + }; let l7_ctx = crate::l7::relay::L7EvalContext { host: host_lc.clone(), @@ -2172,29 +2643,157 @@ async fn handle_forward_proxy( secret_resolver: secret_resolver.clone(), }; - let (target_path, query_params) = crate::l7::rest::parse_target_query(&path) - .unwrap_or_else(|_| (path.clone(), std::collections::HashMap::new())); - let request_info = crate::l7::L7RequestInfo { - action: method.to_string(), - target: target_path, - query_params, + // Canonicalize the request-target. The canonical form is fed to OPA + // AND reassigned to the outer `path` variable so the later call to + // `rewrite_forward_request` writes canonical bytes to the upstream. + // This closes the policy/upstream parser-differential at this site; + // without this reassignment, OPA would evaluate the canonical form + // while the upstream re-normalizes the raw input and dispatches on a + // potentially different path. + let canonicalize_options = crate::l7::path::CanonicalizeOptions { + allow_encoded_slash: route + .configs + .iter() + .any(|snapshot| snapshot.config.allow_encoded_slash), + ..Default::default() }; - - let (allowed, reason) = - crate::l7::relay::evaluate_l7_request(&engine_mutex, &l7_ctx, &request_info) - .unwrap_or_else(|e| { + let query_params = + match crate::l7::path::canonicalize_request_target(&path, &canonicalize_options) { + Ok((canon, query)) => { + upstream_target = match query.as_deref() { + Some(raw_query) if !raw_query.is_empty() => { + format!("{}?{raw_query}", canon.path) + } + _ => canon.path.clone(), + }; + let params = query + .as_deref() + .map_or_else(std::collections::HashMap::new, |q| { + crate::l7::rest::parse_query_params(q).unwrap_or_default() + }); + path = canon.path; + params + } + Err(e) => { let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) .activity(ActivityId::Fail) - .severity(SeverityId::Low) + .severity(SeverityId::Medium) .status(StatusId::Failure) .dst_endpoint(Endpoint::from_domain(&host_lc, port)) - .message(format!("L7 eval failed, denying request: {e}")) + .message(format!( + "FORWARD_L7 rejecting non-canonical request-target: {e}" + )) .build(); ocsf_emit!(event); - (false, format!("L7 evaluation error: {e}")) - }); + respond( + client, + &build_json_error_response( + 400, + "Bad Request", + "invalid_request_target", + "request-target must be canonical", + ), + ) + .await?; + return Ok(()); + } + }; + let Some(l7_config) = select_l7_config_for_path(&route.configs, &path) else { + respond( + client, + &build_json_error_response( + 403, + "Forbidden", + "policy_denied", + &format!("{method} {host_lc}:{port}{path} did not match an L7 endpoint path"), + ), + ) + .await?; + return Ok(()); + }; + let graphql = if l7_config.config.protocol == crate::l7::L7Protocol::Graphql { + let header_end = forward_request_bytes + .windows(4) + .position(|w| w == b"\r\n\r\n") + .map_or(forward_request_bytes.len(), |p| p + 4); + let header_str = std::str::from_utf8(&forward_request_bytes[..header_end]) + .map_err(|_| miette::miette!("Forward GraphQL headers contain invalid UTF-8"))?; + let body_length = crate::l7::rest::parse_body_length(header_str)?; + let mut graphql_request = crate::l7::provider::L7Request { + action: method.to_string(), + target: path.clone(), + query_params: query_params.clone(), + raw_header: forward_request_bytes, + body_length, + }; + let info = match crate::l7::graphql::inspect_graphql_request( + client, + &mut graphql_request, + l7_config.config.graphql_max_body_bytes, + ) + .await + { + Ok(info) => info, + Err(e) => { + let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Fail) + .severity(SeverityId::Medium) + .status(StatusId::Failure) + .dst_endpoint(Endpoint::from_domain(&host_lc, port)) + .message(format!("FORWARD_GRAPHQL_L7 request rejected: {e}")) + .build(); + ocsf_emit!(event); + respond( + client, + &build_json_error_response( + 400, + "Bad Request", + "invalid_graphql_request", + &format!("GraphQL request rejected before policy evaluation: {e}"), + ), + ) + .await?; + return Ok(()); + } + }; + forward_request_bytes = graphql_request.raw_header; + Some(info) + } else { + None + }; + let request_info = crate::l7::L7RequestInfo { + action: method.to_string(), + target: path.clone(), + query_params, + graphql, + }; + + let parse_error_reason = request_info + .graphql + .as_ref() + .and_then(|info| info.error.as_deref()) + .map(|error| format!("GraphQL request rejected: {error}")); + let force_deny = parse_error_reason.is_some(); + let (allowed, reason) = parse_error_reason.map_or_else( + || { + crate::l7::relay::evaluate_l7_request(&tunnel_engine, &l7_ctx, &request_info) + .unwrap_or_else(|e| { + let event = NetworkActivityBuilder::new(crate::ocsf_ctx()) + .activity(ActivityId::Fail) + .severity(SeverityId::Low) + .status(StatusId::Failure) + .dst_endpoint(Endpoint::from_domain(&host_lc, port)) + .message(format!("L7 eval failed, denying request: {e}")) + .build(); + ocsf_emit!(event); + (false, format!("L7 evaluation error: {e}")) + }) + }, + |reason| (false, reason), + ); - let decision_str = match (allowed, l7_config.enforcement) { + let decision_str = match (allowed, l7_config.config.enforcement) { + (_, _) if force_deny => "deny", (true, _) => "allow", (false, crate::l7::EnforcementMode::Audit) => "audit", (false, crate::l7::EnforcementMode::Enforce) => "deny", @@ -2202,13 +2801,8 @@ async fn handle_forward_proxy( { let (action_id, disposition_id, severity) = match decision_str { - "allow" => ( - ActionId::Allowed, - DispositionId::Allowed, - SeverityId::Informational, - ), "deny" => (ActionId::Denied, DispositionId::Blocked, SeverityId::Medium), - "audit" => ( + "allow" | "audit" => ( ActionId::Allowed, DispositionId::Allowed, SeverityId::Informational, @@ -2219,6 +2813,16 @@ async fn handle_forward_proxy( SeverityId::Informational, ), }; + let engine_type = if l7_config.config.protocol == crate::l7::L7Protocol::Graphql { + "l7-graphql" + } else { + "l7" + }; + let message_prefix = if l7_config.config.protocol == crate::l7::L7Protocol::Graphql { + "FORWARD_GRAPHQL_L7" + } else { + "FORWARD_L7" + }; let event = HttpActivityBuilder::new(crate::ocsf_ctx()) .activity(ActivityId::Other) .action(action_id) @@ -2234,16 +2838,16 @@ async fn handle_forward_proxy( Process::from_bypass(&binary_str, &pid_str, &ancestors_str) .with_cmd_line(&cmdline_str), ) - .firewall_rule(policy_str, "l7") + .firewall_rule(policy_str, engine_type) .message(format!( - "FORWARD_L7 {decision_str} {method} {host_lc}:{port}{path} reason={reason}" + "{message_prefix} {decision_str} {method} {host_lc}:{port}{path} reason={reason}" )) .build(); ocsf_emit!(event); } - let effectively_denied = - !allowed && l7_config.enforcement == crate::l7::EnforcementMode::Enforce; + let effectively_denied = force_deny + || (!allowed && l7_config.config.enforcement == crate::l7::EnforcementMode::Enforce); if effectively_denied { emit_denial_simple( @@ -2261,7 +2865,7 @@ async fn handle_forward_proxy( 403, "Forbidden", "policy_denied", - &format!("{method} {host_lc}:{port}{path} denied by L7 policy"), + &format!("{method} {host_lc}:{port}{path} denied by L7 policy: {reason}"), ), ) .await?; @@ -2280,14 +2884,21 @@ async fn handle_forward_proxy( raw_allowed_ips = implicit_allowed_ips_for_ip_host(&host); } - let addrs = if !raw_allowed_ips.is_empty() { - // allowed_ips mode: validate resolved IPs against CIDR allowlist. - match parse_allowed_ips(&raw_allowed_ips) { - Ok(nets) => match resolve_and_check_allowed_ips(&host, port, &nets).await { - Ok(addrs) => addrs, - Err(reason) => { + // The "non-empty" branch is the explicit-allowlist path; reading it first + // matches the policy decision narrative. + #[allow(clippy::if_not_else)] + let addrs = + if !raw_allowed_ips.is_empty() { + // allowed_ips mode: validate resolved IPs against CIDR allowlist. + match parse_allowed_ips(&raw_allowed_ips) { + Ok(nets) => { + match resolve_and_check_allowed_ips(&host, port, &nets, sandbox_entrypoint_pid) + .await { - let event = HttpActivityBuilder::new(crate::ocsf_ctx()) + Ok(addrs) => addrs, + Err(reason) => { + { + let event = HttpActivityBuilder::new(crate::ocsf_ctx()) .activity(ActivityId::Other) .action(ActionId::Denied) .disposition(DispositionId::Blocked) @@ -2309,18 +2920,18 @@ async fn handle_forward_proxy( )) .status_detail(&reason) .build(); - ocsf_emit!(event); - } - emit_denial_simple( - denial_tx, - &host_lc, - port, - &binary_str, - &decision, - &reason, - "ssrf", - ); - respond( + ocsf_emit!(event); + } + emit_denial_simple( + denial_tx, + &host_lc, + port, + &binary_str, + &decision, + &reason, + "ssrf", + ); + respond( client, &build_json_error_response( 403, @@ -2330,12 +2941,13 @@ async fn handle_forward_proxy( ), ) .await?; - return Ok(()); + return Ok(()); + } + } } - }, - Err(reason) => { - { - let event = HttpActivityBuilder::new(crate::ocsf_ctx()) + Err(reason) => { + { + let event = HttpActivityBuilder::new(crate::ocsf_ctx()) .activity(ActivityId::Other) .action(ActionId::Denied) .disposition(DispositionId::Blocked) @@ -2357,39 +2969,39 @@ async fn handle_forward_proxy( )) .status_detail(&reason) .build(); - ocsf_emit!(event); - } - emit_denial_simple( - denial_tx, - &host_lc, - port, - &binary_str, - &decision, - &reason, - "ssrf", - ); - respond( - client, - &build_json_error_response( - 403, - "Forbidden", - "ssrf_denied", - &format!( - "{method} {host_lc}:{port} blocked: invalid allowed_ips in policy" + ocsf_emit!(event); + } + emit_denial_simple( + denial_tx, + &host_lc, + port, + &binary_str, + &decision, + &reason, + "ssrf", + ); + respond( + client, + &build_json_error_response( + 403, + "Forbidden", + "ssrf_denied", + &format!( + "{method} {host_lc}:{port} blocked: invalid allowed_ips in policy" + ), ), - ), - ) - .await?; - return Ok(()); + ) + .await?; + return Ok(()); + } } - } - } else { - // No allowed_ips: reject internal IPs, allow public IPs through. - match resolve_and_reject_internal(&host, port).await { - Ok(addrs) => addrs, - Err(reason) => { - { - let event = HttpActivityBuilder::new(crate::ocsf_ctx()) + } else { + // No allowed_ips: reject internal IPs, allow public IPs through. + match resolve_and_reject_internal(&host, port, sandbox_entrypoint_pid).await { + Ok(addrs) => addrs, + Err(reason) => { + { + let event = HttpActivityBuilder::new(crate::ocsf_ctx()) .activity(ActivityId::Other) .action(ActionId::Denied) .disposition(DispositionId::Blocked) @@ -2411,31 +3023,46 @@ async fn handle_forward_proxy( )) .status_detail(&reason) .build(); - ocsf_emit!(event); + ocsf_emit!(event); + } + emit_denial_simple( + denial_tx, + &host_lc, + port, + &binary_str, + &decision, + &reason, + "ssrf", + ); + respond( + client, + &build_json_error_response( + 403, + "Forbidden", + "ssrf_denied", + &format!("{method} {host_lc}:{port} blocked: internal address"), + ), + ) + .await?; + return Ok(()); } - emit_denial_simple( - denial_tx, - &host_lc, - port, - &binary_str, - &decision, - &reason, - "ssrf", - ); - respond( - client, - &build_json_error_response( - 403, - "Forbidden", - "ssrf_denied", - &format!("{method} {host_lc}:{port} blocked: internal address"), - ), - ) - .await?; - return Ok(()); } - } - }; + }; + + if let Err(e) = forward_generation_guard.ensure_current() { + emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); + respond( + client, + &build_json_error_response( + 403, + "Forbidden", + "policy_denied", + &format!("{method} {host_lc}:{port}{path} not permitted by policy"), + ), + ) + .await?; + return Ok(()); + } // 6. Connect upstream let mut upstream = match TcpStream::connect(addrs.as_slice()).await { @@ -2499,7 +3126,12 @@ async fn handle_forward_proxy( } // 9. Rewrite request and forward to upstream - let rewritten = match rewrite_forward_request(buf, used, &path, secret_resolver.as_deref()) { + let rewritten = match rewrite_forward_request( + &forward_request_bytes, + forward_request_bytes.len(), + &upstream_target, + secret_resolver.as_deref(), + ) { Ok(bytes) => bytes, Err(e) => { warn!( @@ -2521,12 +3153,32 @@ async fn handle_forward_proxy( return Ok(()); } }; - upstream.write_all(&rewritten).await.into_diagnostic()?; - - // 8. Relay remaining traffic bidirectionally (supports streaming) - let _ = tokio::io::copy_bidirectional(client, &mut upstream) - .await - .into_diagnostic()?; + if let Err(e) = forward_generation_guard.ensure_current() { + emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); + respond( + client, + &build_json_error_response( + 403, + "Forbidden", + "policy_denied", + &format!("{method} {host_lc}:{port}{path} not permitted by policy"), + ), + ) + .await?; + return Ok(()); + } + let outcome = relay_rewritten_forward_request( + method, + &path, + rewritten, + client, + &mut upstream, + &forward_generation_guard, + ) + .await?; + if let crate::l7::provider::RelayOutcome::Upgraded { overflow } = outcome { + crate::l7::relay::handle_upgrade(client, &mut upstream, overflow, &host_lc, port).await?; + } Ok(()) } @@ -2588,9 +3240,49 @@ fn is_benign_relay_error(err: &miette::Report) -> bool { } #[cfg(test)] +#[allow( + clippy::needless_raw_string_hashes, + clippy::iter_on_single_items, + clippy::needless_continue, + reason = "Test code: test fixtures and explicit control-flow markers are idiomatic in tests." +)] mod tests { use super::*; - use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; + use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; + + #[test] + fn l7_route_selection_prefers_path_specific_graphql_endpoint() { + let configs = vec![ + L7ConfigSnapshot { + config: crate::l7::L7EndpointConfig { + protocol: crate::l7::L7Protocol::Rest, + path: "/**".to_string(), + tls: crate::l7::TlsMode::Auto, + enforcement: crate::l7::EnforcementMode::Enforce, + graphql_max_body_bytes: crate::l7::graphql::DEFAULT_MAX_BODY_BYTES, + allow_encoded_slash: false, + }, + }, + L7ConfigSnapshot { + config: crate::l7::L7EndpointConfig { + protocol: crate::l7::L7Protocol::Graphql, + path: "/graphql".to_string(), + tls: crate::l7::TlsMode::Auto, + enforcement: crate::l7::EnforcementMode::Enforce, + graphql_max_body_bytes: crate::l7::graphql::DEFAULT_MAX_BODY_BYTES, + allow_encoded_slash: false, + }, + }, + ]; + + let selected = + select_l7_config_for_path(&configs, "/graphql").expect("expected path-specific route"); + assert_eq!(selected.config.protocol, crate::l7::L7Protocol::Graphql); + + let selected = + select_l7_config_for_path(&configs, "/repos/org/repo").expect("expected REST route"); + assert_eq!(selected.config.protocol, crate::l7::L7Protocol::Rest); + } // -- is_internal_ip: IPv4 -- @@ -2748,9 +3440,93 @@ mod tests { // -- resolve_and_reject_internal -- + #[test] + fn test_parse_hosts_file_for_host_handles_comments_invalid_rows_and_case() { + let contents = r#" + # comment + 192.168.1.105 searxng.local searxng + bad-ip ignored.local + 93.184.216.34 Example.Local # trailing comment + ::1 loopback.local + 192.168.1.105 searxng.local + "#; + + let result = parse_hosts_file_for_host(contents, "SEARXNG.LOCAL"); + assert_eq!(result, vec![IpAddr::V4(Ipv4Addr::new(192, 168, 1, 105))]); + + let public = parse_hosts_file_for_host(contents, "example.local"); + assert_eq!(public, vec![IpAddr::V4(Ipv4Addr::new(93, 184, 216, 34))]); + } + + #[test] + fn test_resolve_from_hosts_file_contents_requires_exact_alias_match() { + let contents = "192.168.1.105 searxng.local\n"; + + assert!( + resolve_from_hosts_file_contents(contents, "searxng", 8080).is_empty(), + "partial alias match should not resolve" + ); + + let result = resolve_from_hosts_file_contents(contents, "searxng.local", 8080); + assert_eq!( + result, + vec![SocketAddr::new( + IpAddr::V4(Ipv4Addr::new(192, 168, 1, 105)), + 8080 + )] + ); + } + + #[test] + fn test_resolve_from_hosts_file_contents_public_ip_passes_default_ssrf_check() { + let addrs = + resolve_from_hosts_file_contents("93.184.216.34 example.local\n", "example.local", 80); + assert!(reject_internal_resolved_addrs("example.local", &addrs).is_ok()); + } + + #[test] + fn test_resolve_from_hosts_file_contents_private_ip_requires_allowed_ips() { + let addrs = resolve_from_hosts_file_contents( + "192.168.1.105 searxng.local\n", + "searxng.local", + 8080, + ); + + let err = reject_internal_resolved_addrs("searxng.local", &addrs).unwrap_err(); + assert!( + err.contains("internal address"), + "expected private hosts-file resolution to remain blocked: {err}" + ); + + let nets = parse_allowed_ips(&["192.168.1.105/32".to_string()]).unwrap(); + assert!( + validate_allowed_ips_for_resolved_addrs("searxng.local", 8080, &addrs, &nets).is_ok() + ); + } + + #[test] + fn test_resolve_from_hosts_file_contents_always_blocked_ip_stays_blocked() { + let addrs = + resolve_from_hosts_file_contents("127.0.0.1 loopback.local\n", "loopback.local", 80); + let nets = vec!["127.0.0.0/8".parse::().unwrap()]; + let err = validate_allowed_ips_for_resolved_addrs("loopback.local", 80, &addrs, &nets) + .unwrap_err(); + assert!( + err.contains("always-blocked"), + "expected always-blocked hosts-file resolution to stay blocked: {err}" + ); + } + + #[test] + fn test_resolve_from_hosts_file_contents_returns_empty_without_match() { + let result = + resolve_from_hosts_file_contents("192.168.1.105 searxng.local\n", "missing.local", 80); + assert!(result.is_empty()); + } + #[tokio::test] async fn test_rejects_localhost_resolution() { - let result = resolve_and_reject_internal("localhost", 80).await; + let result = resolve_and_reject_internal("localhost", 80, 0).await; assert!(result.is_err()); let err = result.unwrap_err(); assert!( @@ -2761,7 +3537,7 @@ mod tests { #[tokio::test] async fn test_rejects_loopback_ip_literal() { - let result = resolve_and_reject_internal("127.0.0.1", 443).await; + let result = resolve_and_reject_internal("127.0.0.1", 443, 0).await; assert!(result.is_err()); let err = result.unwrap_err(); assert!( @@ -2772,7 +3548,7 @@ mod tests { #[tokio::test] async fn test_rejects_metadata_ip() { - let result = resolve_and_reject_internal("169.254.169.254", 80).await; + let result = resolve_and_reject_internal("169.254.169.254", 80, 0).await; assert!(result.is_err()); let err = result.unwrap_err(); assert!( @@ -2783,7 +3559,7 @@ mod tests { #[tokio::test] async fn test_dns_failure_returns_error() { - let result = resolve_and_reject_internal("this-host-does-not-exist.invalid", 80).await; + let result = resolve_and_reject_internal("this-host-does-not-exist.invalid", 80, 0).await; assert!(result.is_err()); let err = result.unwrap_err(); assert!( @@ -2792,48 +3568,102 @@ mod tests { ); } - #[test] - fn sanitize_request_headers_strips_auth_and_framing() { - let headers = vec![ - ("authorization".to_string(), "Bearer test".to_string()), - ("x-api-key".to_string(), "secret".to_string()), - ("transfer-encoding".to_string(), "chunked".to_string()), - ("content-length".to_string(), "42".to_string()), - ("content-type".to_string(), "application/json".to_string()), - ("accept".to_string(), "text/event-stream".to_string()), - ]; + #[tokio::test] + async fn inference_interception_applies_router_header_allowlist() { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + use tokio::net::TcpListener; - let kept = sanitize_inference_request_headers(&headers); + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let upstream_addr = listener.local_addr().unwrap(); + let upstream_task = tokio::spawn(async move { + use crate::l7::inference::{ParseResult, try_parse_http_request}; - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("authorization")), - "authorization should be stripped" - ); - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("x-api-key")), - "x-api-key should be stripped" - ); - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("transfer-encoding")), - "transfer-encoding should be stripped" - ); - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("content-length")), - "content-length should be stripped" + let (mut upstream, _) = listener.accept().await.unwrap(); + let mut buf = Vec::new(); + let mut chunk = [0u8; 4096]; + + loop { + let n = upstream.read(&mut chunk).await.unwrap(); + assert!(n > 0, "upstream request closed before request completed"); + buf.extend_from_slice(&chunk[..n]); + + match try_parse_http_request(&buf) { + ParseResult::Complete(_, consumed) => { + upstream + .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok") + .await + .unwrap(); + return String::from_utf8_lossy(&buf[..consumed]).to_string(); + } + ParseResult::Incomplete => continue, + ParseResult::Invalid(reason) => { + panic!("forwarded request should parse cleanly: {reason}"); + } + } + } + }); + + let router = openshell_router::Router::new().unwrap(); + let patterns = crate::l7::inference::default_patterns(); + let ctx = InferenceContext::new( + patterns, + router, + vec![openshell_router::config::ResolvedRoute { + name: "inference.local".to_string(), + endpoint: format!("http://{upstream_addr}"), + model: "meta/llama-3.1-8b-instruct".to_string(), + api_key: "test-api-key".to_string(), + protocols: vec!["openai_chat_completions".to_string()], + auth: openshell_router::config::AuthHeader::Bearer, + default_headers: vec![], + passthrough_headers: vec![ + "openai-organization".to_string(), + "x-model-id".to_string(), + ], + timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, + }], + vec![], ); - assert!( - kept.iter() - .any(|(k, _)| k.eq_ignore_ascii_case("content-type")), - "content-type should be preserved" + + let body = r#"{"model":"ignored","messages":[{"role":"user","content":"hi"}]}"#; + let request = format!( + "POST /v1/chat/completions HTTP/1.1\r\n\ + Host: inference.local\r\n\ + Content-Type: application/json\r\n\ + OpenAI-Organization: org_123\r\n\ + Authorization: Bearer client-key\r\n\ + Cookie: session=abc\r\n\ + Content-Length: {}\r\n\r\n{}", + body.len(), + body, ); + + let (client, mut server) = tokio::io::duplex(65536); + let (mut client_read, mut client_write) = tokio::io::split(client); + + let server_task = + tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); + + client_write.write_all(request.as_bytes()).await.unwrap(); + client_write.shutdown().await.unwrap(); + + let mut response = Vec::new(); + client_read.read_to_end(&mut response).await.unwrap(); + let response_text = String::from_utf8_lossy(&response); + assert!(response_text.starts_with("HTTP/1.1 200")); + + let outcome = server_task.await.unwrap().unwrap(); assert!( - kept.iter().any(|(k, _)| k.eq_ignore_ascii_case("accept")), - "accept should be preserved" + matches!(outcome, InferenceOutcome::Routed), + "expected Routed outcome, got: {outcome:?}" ); + + let forwarded = upstream_task.await.unwrap(); + let forwarded_lc = forwarded.to_ascii_lowercase(); + assert!(forwarded_lc.contains("openai-organization: org_123")); + assert!(forwarded_lc.contains("authorization: bearer test-api-key")); + assert!(!forwarded_lc.contains("authorization: bearer client-key")); + assert!(!forwarded_lc.contains("cookie:")); } // -- router_error_to_http -- @@ -3071,7 +3901,7 @@ mod tests { async fn test_resolve_check_allowed_ips_blocks_loopback() { // Construct nets directly (parse_allowed_ips now rejects always-blocked). let nets = vec!["127.0.0.0/8".parse::().unwrap()]; - let result = resolve_and_check_allowed_ips("127.0.0.1", 80, &nets).await; + let result = resolve_and_check_allowed_ips("127.0.0.1", 80, &nets, 0).await; assert!(result.is_err()); let err = result.unwrap_err(); assert!( @@ -3084,7 +3914,7 @@ mod tests { async fn test_resolve_check_allowed_ips_blocks_metadata() { // Construct nets directly (parse_allowed_ips now rejects always-blocked). let nets = vec!["169.254.0.0/16".parse::().unwrap()]; - let result = resolve_and_check_allowed_ips("169.254.169.254", 80, &nets).await; + let result = resolve_and_check_allowed_ips("169.254.169.254", 80, &nets, 0).await; assert!(result.is_err()); let err = result.unwrap_err(); assert!( @@ -3097,7 +3927,7 @@ mod tests { async fn test_resolve_check_allowed_ips_blocks_unspecified() { // Construct nets directly (parse_allowed_ips now rejects always-blocked). let nets = vec!["0.0.0.0/0".parse::().unwrap()]; - let result = resolve_and_check_allowed_ips("0.0.0.0", 80, &nets).await; + let result = resolve_and_check_allowed_ips("0.0.0.0", 80, &nets, 0).await; assert!(result.is_err()); let err = result.unwrap_err(); assert!( @@ -3110,7 +3940,7 @@ mod tests { async fn test_resolve_check_allowed_ips_rejects_outside_allowlist() { // 8.8.8.8 resolves to a public IP which is NOT in 10.0.0.0/8 let nets = parse_allowed_ips(&["10.0.0.0/8".to_string()]).unwrap(); - let result = resolve_and_check_allowed_ips("dns.google", 443, &nets).await; + let result = resolve_and_check_allowed_ips("dns.google", 443, &nets, 0).await; assert!(result.is_err()); let err = result.unwrap_err(); assert!( @@ -3126,17 +3956,17 @@ mod tests { // Use a public CIDR (parse_allowed_ips now rejects 0.0.0.0/0). let nets = parse_allowed_ips(&["8.8.8.0/24".to_string()]).unwrap(); // K8s API server port - let result = resolve_and_check_allowed_ips("8.8.8.8", 6443, &nets).await; + let result = resolve_and_check_allowed_ips("8.8.8.8", 6443, &nets, 0).await; assert!(result.is_err()); assert!(result.unwrap_err().contains("blocked control-plane port")); // etcd client port - let result = resolve_and_check_allowed_ips("8.8.8.8", 2379, &nets).await; + let result = resolve_and_check_allowed_ips("8.8.8.8", 2379, &nets, 0).await; assert!(result.is_err()); assert!(result.unwrap_err().contains("blocked control-plane port")); // kubelet API port - let result = resolve_and_check_allowed_ips("8.8.8.8", 10250, &nets).await; + let result = resolve_and_check_allowed_ips("8.8.8.8", 10250, &nets, 0).await; assert!(result.is_err()); assert!(result.unwrap_err().contains("blocked control-plane port")); } @@ -3145,7 +3975,7 @@ mod tests { async fn test_resolve_check_allowed_ips_allows_non_control_plane_ports() { // Port 443 should not be blocked by the control-plane port list let nets = parse_allowed_ips(&["8.8.8.0/24".to_string()]).unwrap(); - let result = resolve_and_check_allowed_ips("8.8.8.8", 443, &nets).await; + let result = resolve_and_check_allowed_ips("8.8.8.8", 443, &nets, 0).await; assert!(result.is_ok()); } @@ -3417,6 +4247,59 @@ mod tests { assert!(!result_str.contains("Via: 1.1 openshell-sandbox")); } + #[test] + fn test_rewrite_forward_request_uses_canonical_path_on_the_wire() { + // Regression: the forward-proxy caller must canonicalize first and + // then pass the canonical form to rewrite_forward_request so that + // OPA's policy evaluation and the bytes dispatched to the upstream + // agree. Prior to this guarantee, OPA saw the canonical form while + // the upstream re-normalized the raw path independently, re-opening + // the parser-differential this PR closes. + let raw = b"GET http://host/public/../secret HTTP/1.1\r\nHost: host\r\n\r\n"; + let (canon, _) = crate::l7::path::canonicalize_request_target( + "/public/../secret", + &crate::l7::path::CanonicalizeOptions::default(), + ) + .expect("canonicalization should succeed for the attack payload"); + assert_eq!(canon.path, "/secret"); + + let rewritten = rewrite_forward_request(raw, raw.len(), &canon.path, None) + .expect("rewrite_forward_request should succeed"); + let rewritten_str = String::from_utf8_lossy(&rewritten); + assert!( + rewritten_str.starts_with("GET /secret HTTP/1.1\r\n"), + "outbound request line must use canonical path, got: {rewritten_str:?}" + ); + assert!( + !rewritten_str.contains(".."), + "outbound bytes must not leak the pre-canonical form, got: {rewritten_str:?}" + ); + } + + #[test] + fn test_rewrite_forward_request_preserves_canonical_query_on_the_wire() { + let raw = b"GET http://host/public/../graphql?query=query+Viewer+%7B+viewer+%7B+login+%7D+%7D HTTP/1.1\r\nHost: host\r\n\r\n"; + let (canon, raw_query) = crate::l7::path::canonicalize_request_target( + "/public/../graphql?query=query+Viewer+%7B+viewer+%7B+login+%7D+%7D", + &crate::l7::path::CanonicalizeOptions::default(), + ) + .expect("canonicalization should preserve query separately"); + let upstream_target = match raw_query.as_deref() { + Some(raw_query) if !raw_query.is_empty() => format!("{}?{raw_query}", canon.path), + _ => canon.path, + }; + + let rewritten = rewrite_forward_request(raw, raw.len(), &upstream_target, None) + .expect("rewrite_forward_request should succeed"); + let rewritten_str = String::from_utf8_lossy(&rewritten); + assert!( + rewritten_str.starts_with( + "GET /graphql?query=query+Viewer+%7B+viewer+%7B+login+%7D+%7D HTTP/1.1\r\n" + ), + "outbound request line must preserve canonical query, got: {rewritten_str:?}" + ); + } + #[test] fn test_rewrite_resolves_placeholder_auth_headers() { let (_, resolver) = SecretResolver::from_provider_env( @@ -3432,6 +4315,80 @@ mod tests { assert!(!result_str.contains("openshell:resolve:env:ANTHROPIC_API_KEY")); } + #[tokio::test] + async fn test_forward_relay_guard_blocks_stale_generation_before_upstream_write() { + let policy = include_str!("../data/sandbox-policy.rego"); + let policy_data = "network_policies: {}\n"; + let engine = OpaEngine::from_strings(policy, policy_data).unwrap(); + let guard = engine + .generation_guard(engine.current_generation()) + .unwrap(); + engine.reload(policy, policy_data).unwrap(); + + let raw = b"GET http://host/api HTTP/1.1\r\nHost: host\r\n\r\n"; + let rewritten = + rewrite_forward_request(raw, raw.len(), "/api", None).expect("rewrite should succeed"); + let (mut proxy_to_upstream, mut upstream_side) = tokio::io::duplex(8192); + let (mut _app_side, mut proxy_to_client) = tokio::io::duplex(8192); + + let result = relay_rewritten_forward_request( + "GET", + "/api", + rewritten, + &mut proxy_to_client, + &mut proxy_to_upstream, + &guard, + ) + .await; + assert!( + result.is_err(), + "stale generation must stop forward relay before upstream write" + ); + + drop(proxy_to_upstream); + let mut forwarded = Vec::new(); + upstream_side.read_to_end(&mut forwarded).await.unwrap(); + assert!( + forwarded.is_empty(), + "stale forward request bytes must not reach upstream" + ); + } + + #[tokio::test] + async fn test_forward_relay_rejects_cl_te_smuggling_before_upstream_write() { + let policy = include_str!("../data/sandbox-policy.rego"); + let policy_data = "network_policies: {}\n"; + let engine = OpaEngine::from_strings(policy, policy_data).unwrap(); + let guard = engine + .generation_guard(engine.current_generation()) + .unwrap(); + + let raw = b"POST http://host/api HTTP/1.1\r\nHost: host\r\nContent-Length: 4\r\nTransfer-Encoding: chunked\r\n\r\n0\r\n\r\n"; + let rewritten = + rewrite_forward_request(raw, raw.len(), "/api", None).expect("rewrite should succeed"); + let (mut proxy_to_upstream, mut upstream_side) = tokio::io::duplex(8192); + let (mut _app_side, mut proxy_to_client) = tokio::io::duplex(8192); + + let result = relay_rewritten_forward_request( + "POST", + "/api", + rewritten, + &mut proxy_to_client, + &mut proxy_to_upstream, + &guard, + ) + .await; + assert!(result.is_err(), "forward relay must reject CL/TE ambiguity"); + + drop(proxy_to_upstream); + let mut forwarded = Vec::new(); + upstream_side.read_to_end(&mut forwarded).await.unwrap(); + assert!( + forwarded.is_empty(), + "smuggled forward request bytes must not reach upstream" + ); + } + // --- Forward proxy SSRF defence tests --- // // The forward proxy handler uses the same SSRF logic as the CONNECT path: @@ -3444,7 +4401,7 @@ mod tests { async fn test_forward_public_ip_allowed_without_allowed_ips() { // Public IPs (e.g. dns.google -> 8.8.8.8) should pass through // resolve_and_reject_internal without needing allowed_ips. - let result = resolve_and_reject_internal("dns.google", 80).await; + let result = resolve_and_reject_internal("dns.google", 80, 0).await; assert!( result.is_ok(), "Public IP should be allowed without allowed_ips: {result:?}" @@ -3464,7 +4421,7 @@ mod tests { #[tokio::test] async fn test_forward_private_ip_rejected_without_allowed_ips() { // Private IP literals should be rejected by resolve_and_reject_internal. - let result = resolve_and_reject_internal("10.0.0.1", 80).await; + let result = resolve_and_reject_internal("10.0.0.1", 80, 0).await; assert!( result.is_err(), "Private IP should be rejected without allowed_ips" @@ -3480,7 +4437,7 @@ mod tests { async fn test_forward_private_ip_accepted_with_allowed_ips() { // Private IP with matching allowed_ips should pass through. let nets = parse_allowed_ips(&["10.0.0.0/8".to_string()]).unwrap(); - let result = resolve_and_check_allowed_ips("10.0.0.1", 80, &nets).await; + let result = resolve_and_check_allowed_ips("10.0.0.1", 80, &nets, 0).await; assert!( result.is_ok(), "Private IP with matching allowed_ips should be accepted: {result:?}" @@ -3491,7 +4448,7 @@ mod tests { async fn test_forward_private_ip_rejected_with_wrong_allowed_ips() { // Private IP not in allowed_ips should be rejected. let nets = parse_allowed_ips(&["192.168.0.0/16".to_string()]).unwrap(); - let result = resolve_and_check_allowed_ips("10.0.0.1", 80, &nets).await; + let result = resolve_and_check_allowed_ips("10.0.0.1", 80, &nets, 0).await; assert!( result.is_err(), "Private IP not in allowed_ips should be rejected" @@ -3508,7 +4465,7 @@ mod tests { // Loopback addresses are always blocked, even if in allowed_ips. // Construct nets directly (parse_allowed_ips now rejects always-blocked). let nets = vec!["127.0.0.0/8".parse::().unwrap()]; - let result = resolve_and_check_allowed_ips("127.0.0.1", 80, &nets).await; + let result = resolve_and_check_allowed_ips("127.0.0.1", 80, &nets, 0).await; assert!(result.is_err(), "Loopback should be always blocked"); let err = result.unwrap_err(); assert!( @@ -3522,7 +4479,7 @@ mod tests { // Link-local / cloud metadata addresses are always blocked. // Construct nets directly (parse_allowed_ips now rejects always-blocked). let nets = vec!["169.254.0.0/16".parse::().unwrap()]; - let result = resolve_and_check_allowed_ips("169.254.169.254", 80, &nets).await; + let result = resolve_and_check_allowed_ips("169.254.169.254", 80, &nets, 0).await; assert!(result.is_err(), "Link-local should be always blocked"); let err = result.unwrap_err(); assert!( @@ -3699,4 +4656,286 @@ mod tests { let body_start = resp_str.find("\r\n\r\n").unwrap() + 4; assert_eq!(resp_str[body_start..].len(), cl); } + + /// End-to-end regression for the `docker cp` hot-swap hazard that + /// motivated `binary_path()` stripping the kernel's `" (deleted)"` + /// suffix (PR #844). + /// + /// Before the strip, the identity-resolution chain inside + /// `evaluate_opa_tcp` failed with `"Failed to stat + /// /opt/openshell/bin/openshell-sandbox (deleted)"` because + /// `BinaryIdentityCache::verify_or_cache()` tried to `metadata()` the + /// tainted path. That masked the real security signal: a live process + /// was now bound to a *different* binary on disk than the one that was + /// TOFU-cached. After the strip, `binary_path()` returns a path that + /// stats fine, the cache rehashes the new bytes, and the hash mismatch + /// surfaces as a `Binary integrity violation` error — the contract this + /// PR is trying to establish. + /// + /// Test shape (from the review comment on the initial PR): + /// 1. Start a `TcpListener` in the test process. + /// 2. Copy `/bin/bash` to a temp path we control. + /// 3. Prime `BinaryIdentityCache` with that temp binary's hash. + /// 4. Spawn the temp bash as a child with a `/dev/tcp` one-liner that + /// opens a real TCP connection to the listener and holds it open. + /// 5. Accept the connection on the listener side and capture the peer's + /// ephemeral port — that's what `resolve_process_identity` uses to + /// walk `/proc/net/tcp` back to the child PID. + /// 6. Overwrite the temp bash on disk with different bytes to simulate + /// a `docker cp` hot-swap. The running child is unaffected (it still + /// executes from its in-memory image), but `/proc//exe` will + /// now readlink to `" (deleted)"` OR the overwritten file, depending + /// on whether the filesystem reused the inode. + /// 7. Call `resolve_process_identity` and assert: + /// - the error reason contains `"Binary integrity violation"` (the + /// cache detected the tampered on-disk bytes), and + /// - the error reason does NOT contain `"Failed to stat"` or + /// `"(deleted)"` (the old pre-strip failure mode). + #[cfg(target_os = "linux")] + #[test] + fn resolve_process_identity_surfaces_binary_integrity_violation_on_hot_swap() { + use crate::identity::BinaryIdentityCache; + use std::io::Read; + use std::net::TcpListener; + use std::os::unix::fs::PermissionsExt; + use std::process::{Command, Stdio}; + use std::time::Duration; + + // Skip if /bin/bash is not present (e.g. minimal containers). + if !std::path::Path::new("/bin/bash").exists() { + eprintln!("skipping: /bin/bash not available"); + return; + } + + // 1. Start a listener on loopback. + let listener = TcpListener::bind("127.0.0.1:0").expect("bind"); + let listener_port = listener.local_addr().unwrap().port(); + + // 2. Copy /bin/bash to a temp path. + let tmp = tempfile::TempDir::new().unwrap(); + let bash_v1 = tmp.path().join("hotswap-bash"); + std::fs::copy("/bin/bash", &bash_v1).expect("copy bash"); + std::fs::set_permissions(&bash_v1, std::fs::Permissions::from_mode(0o755)).unwrap(); + + // 3. Prime the cache with the v1 hash of the temp bash. + let cache = BinaryIdentityCache::new(); + let v1_hash = cache + .verify_or_cache(&bash_v1) + .expect("prime cache with v1 bash hash"); + assert!(!v1_hash.is_empty()); + + // 4. Spawn the temp bash with a /dev/tcp one-liner that opens a real + // connection to the listener and sleeps to keep it open. The + // `read -t` blocks on stdin so the shell stays resident. + let script = format!("exec 3<>/dev/tcp/127.0.0.1/{listener_port}; sleep 30 <&3"); + let mut child = Command::new(&bash_v1) + .arg("-c") + .arg(&script) + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .expect("spawn hotswap-bash child"); + + // 5. Accept on the listener side, capture the peer port. + listener.set_nonblocking(false).expect("blocking listener"); + let (mut stream, peer_addr) = match listener.accept() { + Ok(pair) => pair, + Err(e) => { + let _ = child.kill(); + let _ = child.wait(); + panic!("failed to accept child connection: {e}"); + } + }; + let peer_port = peer_addr.port(); + // Drain any spurious data; we just need the socket open. + stream + .set_read_timeout(Some(Duration::from_millis(50))) + .ok(); + let mut buf = [0u8; 16]; + let _ = stream.read(&mut buf); + + // Give the kernel a moment so /proc//net/tcp and + // /proc//fd/ both reflect the ESTABLISHED socket. + std::thread::sleep(Duration::from_millis(50)); + + // 6. Simulate `docker cp`: unlink the running binary and create a + // fresh file with different bytes at the same path. Writing + // in place via O_TRUNC is rejected by the kernel with ETXTBSY + // because the inode is still being executed. Unlink is cheap: + // the inode persists in memory via the child's exec mapping, + // so the child keeps running, but a new inode now lives at + // `bash_v1` with a different SHA-256. + std::fs::remove_file(&bash_v1).expect("unlink running bash_v1"); + let tampered_bytes = b"#!/bin/sh\n# tampered bash v2 from hotswap test\nexit 0\n"; + std::fs::write(&bash_v1, tampered_bytes).expect("write replacement bytes"); + + // 7. Resolve identity through the real helper and assert the + // contract: we want "Binary integrity violation", not + // "Failed to stat ... (deleted)". + let test_pid = std::process::id(); + let result = resolve_process_identity(test_pid, peer_port, &cache); + + // Always clean up the child before asserting so a failure doesn't + // leak a sleeping process across test runs. + let _ = child.kill(); + let _ = child.wait(); + + match result { + Ok(_) => panic!( + "resolve_process_identity unexpectedly succeeded after hot-swap; \ + the cache should have detected the tampered on-disk bytes" + ), + Err(err) => { + assert!( + err.reason.contains("Binary integrity violation"), + "expected 'Binary integrity violation' error, got: {}", + err.reason + ); + assert!( + !err.reason.contains("Failed to stat"), + "pre-PR-#844 failure mode leaked: {}", + err.reason + ); + assert!( + !err.reason.contains("(deleted)"), + "resolved path still contains '(deleted)' suffix: {}", + err.reason + ); + // The binary field should be populated — we did resolve a + // path before failing. + assert!( + err.binary.is_some(), + "expected resolved binary path on integrity failure" + ); + if let Some(path) = &err.binary { + assert!( + !path.to_string_lossy().contains("(deleted)"), + "resolved binary path still tainted: {}", + path.display() + ); + } + } + } + } + + #[cfg(target_os = "linux")] + #[test] + fn resolve_process_identity_denies_fork_exec_shared_socket_ambiguity() { + use crate::identity::BinaryIdentityCache; + use std::ffi::CString; + use std::net::{TcpListener, TcpStream}; + use std::os::fd::AsRawFd; + use std::time::{Duration, Instant}; + + if !std::path::Path::new("/bin/sleep").exists() { + eprintln!("skipping: /bin/sleep not available"); + return; + } + + let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); + let listener_port = listener.local_addr().unwrap().port(); + let stream = TcpStream::connect(("127.0.0.1", listener_port)).expect("connect"); + let peer_port = stream.local_addr().unwrap().port(); + let (_accepted, _) = listener.accept().expect("accept"); + + let fd = stream.as_raw_fd(); + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + unsafe { + let flags = libc::fcntl(fd, libc::F_GETFD); + assert!(flags >= 0, "F_GETFD failed"); + assert_eq!( + libc::fcntl(fd, libc::F_SETFD, flags & !libc::FD_CLOEXEC), + 0, + "F_SETFD failed" + ); + } + + let sleep_path = CString::new("/bin/sleep").unwrap(); + let arg0 = CString::new("sleep").unwrap(); + let arg1 = CString::new("30").unwrap(); + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + let child_pid = unsafe { libc::fork() }; + assert!(child_pid >= 0, "fork failed"); + if child_pid == 0 { + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + unsafe { + libc::execl( + sleep_path.as_ptr(), + arg0.as_ptr(), + arg1.as_ptr(), + std::ptr::null::(), + ); + libc::_exit(127); + } + } + + let deadline = Instant::now() + Duration::from_secs(2); + loop { + if let Ok(link) = std::fs::read_link(format!("/proc/{child_pid}/exe")) + && link.to_string_lossy().contains("sleep") + { + break; + } + assert!( + Instant::now() < deadline, + "child pid {child_pid} did not exec into sleep within 2s" + ); + std::thread::sleep(Duration::from_millis(20)); + } + + let cache = BinaryIdentityCache::new(); + + // Resolve with a brief retry loop — under heavy CI load the child's + // procfs entry can momentarily fail to resolve even though the loop + // above just verified `/proc//exe` pointed at `sleep`. Retry a + // few times before declaring failure so the test is not flaky. + let mut result = resolve_process_identity(std::process::id(), peer_port, &cache); + for _ in 0..5 { + match &result { + Err(err) + if err.reason.contains("No such file or directory") + || err.reason.contains("os error 2") => + { + std::thread::sleep(Duration::from_millis(50)); + result = resolve_process_identity(std::process::id(), peer_port, &cache); + } + _ => break, + } + } + + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] + unsafe { + libc::kill(child_pid, libc::SIGKILL); + libc::waitpid(child_pid, std::ptr::null_mut(), 0); + } + + match result { + Ok(identity) => panic!( + "resolve_process_identity unexpectedly succeeded for shared socket owned by PID {}", + identity.binary_pid + ), + Err(err) => { + assert!( + err.reason.contains("ambiguous shared socket ownership"), + "expected ambiguous socket ownership error, got: {}", + err.reason + ); + assert!( + err.reason.contains(&std::process::id().to_string()), + "error should include parent PID; got: {}", + err.reason + ); + assert!( + err.reason.contains(&child_pid.to_string()), + "error should include child PID; got: {}", + err.reason + ); + } + } + } } diff --git a/crates/openshell-sandbox/src/sandbox/linux/landlock.rs b/crates/openshell-sandbox/src/sandbox/linux/landlock.rs index b982c52385..214fc700af 100644 --- a/crates/openshell-sandbox/src/sandbox/linux/landlock.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/landlock.rs @@ -95,7 +95,7 @@ pub struct PreparedRuleset { compatibility: LandlockCompatibility, } -/// Phase 1: Open PathFds and build the Landlock ruleset **as root**. +/// Phase 1: Open `PathFds` and build the Landlock ruleset **as root**. /// /// This must run before `drop_privileges()` so that `PathFd::new()` can open /// paths that are only accessible to root (e.g. mode 700 directories). @@ -272,6 +272,7 @@ pub fn enforce(prepared: PreparedRuleset) -> Result<()> { /// Legacy single-phase apply. Kept for non-Linux platforms and tests. /// On Linux, callers should use [`prepare`] + [`enforce`] for correct /// privilege ordering. +#[allow(dead_code)] // Retained for backward compat; live callers use prepare+enforce. pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> { if let Some(prepared) = prepare(policy, workdir)? { enforce(prepared)?; diff --git a/crates/openshell-sandbox/src/sandbox/linux/mod.rs b/crates/openshell-sandbox/src/sandbox/linux/mod.rs index 988aff1cab..848ab1e3b7 100644 --- a/crates/openshell-sandbox/src/sandbox/linux/mod.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/mod.rs @@ -13,7 +13,7 @@ use std::path::PathBuf; use std::sync::Once; /// Opaque handle to a prepared-but-not-yet-enforced sandbox. -/// Holds the Landlock ruleset with PathFds opened as root. +/// Holds the Landlock ruleset with `PathFds` opened as root. pub struct PreparedSandbox { landlock: Option, policy: SandboxPolicy, @@ -21,7 +21,7 @@ pub struct PreparedSandbox { /// Phase 1: Prepare sandbox restrictions **as root** (before `drop_privileges`). /// -/// Opens Landlock PathFds while the process still has root privileges, +/// Opens Landlock `PathFds` while the process still has root privileges, /// ensuring paths like mode-700 directories are accessible. pub fn prepare(policy: &SandboxPolicy, workdir: Option<&str>) -> Result { let landlock = landlock::prepare(policy, workdir)?; @@ -43,8 +43,14 @@ pub fn enforce(prepared: PreparedSandbox) -> Result<()> { Ok(()) } +/// Apply the supervisor seccomp prelude after privileged bootstrap completes. +pub fn apply_supervisor_prelude() -> Result<()> { + seccomp::apply_supervisor_prelude() +} + /// Legacy single-phase apply. Kept for backward compatibility. /// New callers should use [`prepare`] + [`enforce`] for correct privilege ordering. +#[allow(dead_code)] // Retained for backward compat; live callers use prepare+enforce. pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> { landlock::apply(policy, workdir)?; seccomp::apply(policy)?; @@ -67,12 +73,12 @@ pub fn log_sandbox_readiness(policy: &SandboxPolicy, workdir: Option<&str>) { let mut read_write = policy.filesystem.read_write.clone(); let read_only = &policy.filesystem.read_only; - if policy.filesystem.include_workdir { - if let Some(dir) = workdir { - let workdir_path = PathBuf::from(dir); - if !read_write.contains(&workdir_path) { - read_write.push(workdir_path); - } + if policy.filesystem.include_workdir + && let Some(dir) = workdir + { + let workdir_path = PathBuf::from(dir); + if !read_write.contains(&workdir_path) { + read_write.push(workdir_path); } } @@ -91,74 +97,71 @@ pub fn log_sandbox_readiness(policy: &SandboxPolicy, workdir: Option<&str>) { } let availability = landlock::probe_availability(); - match &availability { - landlock::LandlockAvailability::Available { abi } => { - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(crate::ocsf_ctx()) - .severity(openshell_ocsf::SeverityId::Informational) - .status(openshell_ocsf::StatusId::Success) - .state(openshell_ocsf::StateId::Enabled, "probed") - .message(format!( - "Landlock filesystem sandbox available \ - [abi:v{abi} compat:{:?} ro:{} rw:{}]", - policy.landlock.compatibility, - read_only.len(), - read_write.len(), - )) - .build() - ); - } - _ => { - // Landlock is NOT available — this is the critical log that was - // previously invisible because it only fired inside pre_exec. - let is_best_effort = matches!( - policy.landlock.compatibility, - crate::policy::LandlockCompatibility::BestEffort - ); - let (desc, msg) = if is_best_effort { - ( - format!( - "Sandbox will run WITHOUT filesystem restrictions: {availability}. \ - Policy requests {total_paths} path rule(s) \ - (ro:{} rw:{}) but Landlock cannot enforce them. \ - Set landlock.compatibility to 'hard_requirement' to make this fatal.", - read_only.len(), - read_write.len(), - ), - format!( - "Landlock filesystem sandbox unavailable (best_effort, degraded): {availability}" - ), - ) - } else { - ( - format!( - "Landlock is unavailable: {availability}. \ - Policy requires {total_paths} path rule(s) \ - (ro:{} rw:{}) with hard_requirement — sandbox startup will fail.", - read_only.len(), - read_write.len(), - ), - format!( - "Landlock filesystem sandbox unavailable (hard_requirement, will fail): {availability}" - ), - ) - }; - openshell_ocsf::ocsf_emit!( - openshell_ocsf::DetectionFindingBuilder::new(crate::ocsf_ctx()) - .activity(openshell_ocsf::ActivityId::Open) - .severity(openshell_ocsf::SeverityId::High) - .confidence(openshell_ocsf::ConfidenceId::High) - .is_alert(true) - .finding_info( - openshell_ocsf::FindingInfo::new( - "landlock-unavailable", - "Landlock Filesystem Sandbox Unavailable", - ) - .with_desc(&desc), + if let landlock::LandlockAvailability::Available { abi } = &availability { + openshell_ocsf::ocsf_emit!( + openshell_ocsf::ConfigStateChangeBuilder::new(crate::ocsf_ctx()) + .severity(openshell_ocsf::SeverityId::Informational) + .status(openshell_ocsf::StatusId::Success) + .state(openshell_ocsf::StateId::Enabled, "probed") + .message(format!( + "Landlock filesystem sandbox available \ + [abi:v{abi} compat:{:?} ro:{} rw:{}]", + policy.landlock.compatibility, + read_only.len(), + read_write.len(), + )) + .build() + ); + } else { + // Landlock is NOT available — this is the critical log that was + // previously invisible because it only fired inside pre_exec. + let is_best_effort = matches!( + policy.landlock.compatibility, + crate::policy::LandlockCompatibility::BestEffort + ); + let (desc, msg) = if is_best_effort { + ( + format!( + "Sandbox will run WITHOUT filesystem restrictions: {availability}. \ + Policy requests {total_paths} path rule(s) \ + (ro:{} rw:{}) but Landlock cannot enforce them. \ + Set landlock.compatibility to 'hard_requirement' to make this fatal.", + read_only.len(), + read_write.len(), + ), + format!( + "Landlock filesystem sandbox unavailable (best_effort, degraded): {availability}" + ), + ) + } else { + ( + format!( + "Landlock is unavailable: {availability}. \ + Policy requires {total_paths} path rule(s) \ + (ro:{} rw:{}) with hard_requirement — sandbox startup will fail.", + read_only.len(), + read_write.len(), + ), + format!( + "Landlock filesystem sandbox unavailable (hard_requirement, will fail): {availability}" + ), + ) + }; + openshell_ocsf::ocsf_emit!( + openshell_ocsf::DetectionFindingBuilder::new(crate::ocsf_ctx()) + .activity(openshell_ocsf::ActivityId::Open) + .severity(openshell_ocsf::SeverityId::High) + .confidence(openshell_ocsf::ConfidenceId::High) + .is_alert(true) + .finding_info( + openshell_ocsf::FindingInfo::new( + "landlock-unavailable", + "Landlock Filesystem Sandbox Unavailable", ) - .message(msg) - .build() - ); - } + .with_desc(&desc), + ) + .message(msg) + .build() + ); } } diff --git a/crates/openshell-sandbox/src/sandbox/linux/netns.rs b/crates/openshell-sandbox/src/sandbox/linux/netns.rs index 37d11f0c3e..019036e533 100644 --- a/crates/openshell-sandbox/src/sandbox/linux/netns.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/netns.rs @@ -10,14 +10,22 @@ use miette::{IntoDiagnostic, Result}; use std::net::IpAddr; use std::os::unix::io::RawFd; +use std::path::Path; use std::process::Command; -use tracing::{debug, info, warn}; +use tracing::{debug, warn}; use uuid::Uuid; /// Default subnet for sandbox networking. const SUBNET_PREFIX: &str = "10.200.0"; const HOST_IP_SUFFIX: u8 = 1; const SANDBOX_IP_SUFFIX: u8 = 2; +const IP_SEARCH_PATHS: &[&str] = &["/usr/sbin/ip", "/sbin/ip", "/usr/bin/ip", "/bin/ip"]; +const NSENTER_SEARCH_PATHS: &[&str] = &[ + "/usr/bin/nsenter", + "/bin/nsenter", + "/usr/sbin/nsenter", + "/sbin/nsenter", +]; /// Handle to a network namespace with veth pair. /// @@ -211,6 +219,8 @@ impl NetworkNamespace { if let Some(fd) = self.ns_fd { debug!(namespace = %self.name, "Entering network namespace via setns"); // SAFETY: setns is safe to call after fork, before exec + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] let result = unsafe { libc::setns(fd, libc::CLONE_NEWNET) }; if result != 0 { return Err(miette::miette!( @@ -235,7 +245,7 @@ impl NetworkNamespace { /// Install iptables rules for bypass detection inside the namespace. /// /// Sets up OUTPUT chain rules that: - /// 1. ACCEPT traffic destined for the proxy (host_ip:proxy_port) + /// 1. ACCEPT traffic destined for the proxy (`host_ip:proxy_port`) /// 2. ACCEPT loopback traffic /// 3. ACCEPT established/related connections (response packets) /// 4. LOG + REJECT all other TCP/UDP traffic (bypass attempts) @@ -251,22 +261,19 @@ impl NetworkNamespace { /// diagnostic logging. pub fn install_bypass_rules(&self, proxy_port: u16) -> Result<()> { // Check if iptables is available before attempting to install rules. - let iptables_path = match find_iptables() { - Some(path) => path, - None => { - openshell_ocsf::ocsf_emit!(openshell_ocsf::ConfigStateChangeBuilder::new( - crate::ocsf_ctx() - ) - .severity(openshell_ocsf::SeverityId::Medium) - .status(openshell_ocsf::StatusId::Failure) - .state(openshell_ocsf::StateId::Disabled, "degraded") - .message(format!( - "iptables not found; bypass detection rules will not be installed [ns:{}]", - self.name - )) - .build()); - return Ok(()); - } + let Some(iptables_path) = find_iptables() else { + openshell_ocsf::ocsf_emit!( + openshell_ocsf::ConfigStateChangeBuilder::new(crate::ocsf_ctx()) + .severity(openshell_ocsf::SeverityId::Medium) + .status(openshell_ocsf::StatusId::Failure) + .state(openshell_ocsf::StateId::Disabled, "degraded") + .message(format!( + "iptables not found; bypass detection rules will not be installed [ns:{}]", + self.name + )) + .build() + ); + return Ok(()); }; let host_ip_str = self.host_ip.to_string(); @@ -299,20 +306,20 @@ impl NetworkNamespace { // Install IPv6 rules — best-effort. // Skip the proxy ACCEPT rule for IPv6 since the proxy address is IPv4. - if let Some(ip6_path) = find_ip6tables(&iptables_path) { - if let Err(e) = self.install_bypass_rules_for_v6(&ip6_path, &log_prefix) { - openshell_ocsf::ocsf_emit!(openshell_ocsf::ConfigStateChangeBuilder::new( - crate::ocsf_ctx() - ) - .severity(openshell_ocsf::SeverityId::Low) - .status(openshell_ocsf::StatusId::Failure) - .state(openshell_ocsf::StateId::Other, "degraded") - .message(format!( - "Failed to install IPv6 bypass detection rules (non-fatal) [ns:{}]: {e}", - self.name - )) - .build()); - } + if let Some(ip6_path) = find_ip6tables(&iptables_path) + && let Err(e) = self.install_bypass_rules_for_v6(&ip6_path, &log_prefix) + { + openshell_ocsf::ocsf_emit!( + openshell_ocsf::ConfigStateChangeBuilder::new(crate::ocsf_ctx()) + .severity(openshell_ocsf::SeverityId::Low) + .status(openshell_ocsf::StatusId::Failure) + .state(openshell_ocsf::StateId::Other, "degraded") + .message(format!( + "Failed to install IPv6 bypass detection rules (non-fatal) [ns:{}]: {e}", + self.name + )) + .build() + ); } openshell_ocsf::ocsf_emit!( @@ -662,14 +669,19 @@ impl Drop for NetworkNamespace { /// Run an `ip` command on the host. fn run_ip(args: &[&str]) -> Result<()> { - debug!(command = %format!("ip {}", args.join(" ")), "Running ip command"); + let ip_path = find_trusted_binary("ip", IP_SEARCH_PATHS)?; - let output = Command::new("ip").args(args).output().into_diagnostic()?; + debug!(command = %format!("{ip_path} {}", args.join(" ")), "Running ip command"); + + let output = Command::new(ip_path) + .args(args) + .output() + .into_diagnostic()?; if !output.status.success() { let stderr = String::from_utf8_lossy(&output.stderr); return Err(miette::miette!( - "ip {} failed: {}", + "{ip_path} {} failed: {}", args.join(" "), stderr.trim() )); @@ -678,14 +690,31 @@ fn run_ip(args: &[&str]) -> Result<()> { Ok(()) } -/// Run an `ip netns exec` command inside a namespace. +/// Run an `ip` command inside a network namespace via `nsenter --net=`. +/// +/// We use `nsenter` instead of `ip netns exec` because `ip netns exec` +/// remounts `/sys` to reflect the target namespace's sysfs entries. That +/// sysfs remount requires real `CAP_SYS_ADMIN` in the host user namespace, +/// which is unavailable in rootless container runtimes (e.g. rootless +/// Podman). `nsenter --net=` enters only the network namespace without +/// changing the mount namespace, avoiding the sysfs remount entirely. +/// The supervisor's operations (addr add, link set, route add) are all +/// netlink-based and do not need sysfs access. fn run_ip_netns(netns: &str, args: &[&str]) -> Result<()> { - let mut full_args = vec!["netns", "exec", netns, "ip"]; + let ip_path = find_trusted_binary("ip", IP_SEARCH_PATHS)?; + let nsenter_path = find_trusted_binary("nsenter", NSENTER_SEARCH_PATHS)?; + let ns_path = format!("/var/run/netns/{netns}"); + let net_flag = format!("--net={ns_path}"); + + let mut full_args = vec![net_flag.as_str(), "--", ip_path]; full_args.extend(args); - debug!(command = %format!("ip {}", full_args.join(" ")), "Running ip netns exec command"); + debug!( + command = %format!("{nsenter_path} {}", full_args.join(" ")), + "Running ip in namespace via nsenter" + ); - let output = Command::new("ip") + let output = Command::new(nsenter_path) .args(&full_args) .output() .into_diagnostic()?; @@ -693,8 +722,8 @@ fn run_ip_netns(netns: &str, args: &[&str]) -> Result<()> { if !output.status.success() { let stderr = String::from_utf8_lossy(&output.stderr); return Err(miette::miette!( - "ip netns exec {} ip {} failed: {}", - netns, + "{nsenter_path} --net={} {ip_path} {} failed: {}", + ns_path, args.join(" "), stderr.trim() )); @@ -703,17 +732,24 @@ fn run_ip_netns(netns: &str, args: &[&str]) -> Result<()> { Ok(()) } -/// Run an iptables command inside a network namespace. +/// Run an iptables command inside a network namespace via `nsenter --net=`. +/// +/// Uses `nsenter` instead of `ip netns exec` to avoid the sysfs remount +/// that fails in rootless container runtimes. See `run_ip_netns` for details. fn run_iptables_netns(netns: &str, iptables_cmd: &str, args: &[&str]) -> Result<()> { - let mut full_args = vec!["netns", "exec", netns, iptables_cmd]; + let nsenter_path = find_trusted_binary("nsenter", NSENTER_SEARCH_PATHS)?; + let ns_path = format!("/var/run/netns/{netns}"); + let net_flag = format!("--net={ns_path}"); + + let mut full_args = vec![net_flag.as_str(), "--", iptables_cmd]; full_args.extend(args); debug!( - command = %format!("ip {}", full_args.join(" ")), - "Running iptables in namespace" + command = %format!("{nsenter_path} {}", full_args.join(" ")), + "Running iptables in namespace via nsenter" ); - let output = Command::new("ip") + let output = Command::new(nsenter_path) .args(&full_args) .output() .into_diagnostic()?; @@ -721,8 +757,8 @@ fn run_iptables_netns(netns: &str, iptables_cmd: &str, args: &[&str]) -> Result< if !output.status.success() { let stderr = String::from_utf8_lossy(&output.stderr); return Err(miette::miette!( - "ip netns exec {} {} failed: {}", - netns, + "{nsenter_path} --net={} {} failed: {}", + ns_path, iptables_cmd, stderr.trim() )); @@ -737,13 +773,29 @@ fn run_iptables_netns(netns: &str, iptables_cmd: &str, args: &[&str]) -> Result< const IPTABLES_SEARCH_PATHS: &[&str] = &["/usr/sbin/iptables", "/sbin/iptables", "/usr/bin/iptables"]; -/// Returns true if xt extension modules (e.g. xt_comment) cannot be used +fn find_trusted_binary<'a>(name: &str, paths: &'a [&str]) -> Result<&'a str> { + paths + .iter() + .copied() + .find(|path| { + let path = Path::new(path); + path.is_absolute() && path.is_file() + }) + .ok_or_else(|| { + miette::miette!( + "trusted {name} helper not found; checked {}", + paths.join(", ") + ) + }) +} + +/// Returns true if xt extension modules (e.g. `xt_comment`) cannot be used /// via the given iptables binary. /// -/// Some kernels have nf_tables but lack the nft_compat bridge that allows -/// xt extension modules to be used through the nf_tables path (e.g. Jetson +/// Some kernels have `nf_tables` but lack the `nft_compat` bridge that allows +/// xt extension modules to be used through the `nf_tables` path (e.g. Jetson /// Linux 5.15-tegra). This probe detects that condition by attempting to -/// insert a rule using the xt_comment extension. If it fails, xt extensions +/// insert a rule using the `xt_comment` extension. If it fails, xt extensions /// are unavailable and the caller should fall back to iptables-legacy. fn xt_extensions_unavailable(iptables_path: &str) -> bool { // Create a temporary probe chain. If this fails (e.g. no CAP_NET_ADMIN), @@ -751,8 +803,7 @@ fn xt_extensions_unavailable(iptables_path: &str) -> bool { let created = Command::new(iptables_path) .args(["-t", "filter", "-N", "_xt_probe"]) .output() - .map(|o| o.status.success()) - .unwrap_or(false); + .is_ok_and(|o| o.status.success()); if !created { return false; @@ -774,8 +825,7 @@ fn xt_extensions_unavailable(iptables_path: &str) -> bool { "ACCEPT", ]) .output() - .map(|o| o.status.success()) - .unwrap_or(false); + .is_ok_and(|o| o.status.success()); // Clean up — best-effort, ignore failures. let _ = Command::new(iptables_path) @@ -808,12 +858,12 @@ fn xt_extensions_unavailable(iptables_path: &str) -> bool { fn find_iptables() -> Option { let standard_path = IPTABLES_SEARCH_PATHS .iter() - .find(|path| std::path::Path::new(path).exists()) + .find(|path| Path::new(path).exists()) .copied()?; if xt_extensions_unavailable(standard_path) { let legacy_path = standard_path.replace("iptables", "iptables-legacy"); - if std::path::Path::new(&legacy_path).exists() { + if Path::new(&legacy_path).exists() { debug!( legacy = legacy_path, "xt extensions unavailable; using iptables-legacy" @@ -828,7 +878,7 @@ fn find_iptables() -> Option { /// Find the ip6tables binary path, deriving it from the iptables location. fn find_ip6tables(iptables_path: &str) -> Option { let ip6_path = iptables_path.replace("iptables", "ip6tables"); - if std::path::Path::new(&ip6_path).exists() { + if Path::new(&ip6_path).exists() { Some(ip6_path) } else { None @@ -838,10 +888,32 @@ fn find_ip6tables(iptables_path: &str) -> Option { #[cfg(test)] mod tests { use super::*; + use std::fs; // These tests require root and network namespace support // Run with: sudo cargo test -- --ignored + #[test] + fn find_trusted_binary_uses_absolute_existing_file() { + let tempdir = tempfile::tempdir().unwrap(); + let helper = tempdir.path().join("ip"); + fs::write(&helper, b"test helper").unwrap(); + let helper = helper.to_str().unwrap(); + + assert_eq!( + find_trusted_binary("ip", &["relative-ip", "/missing/ip", helper]).unwrap(), + helper + ); + } + + #[test] + fn find_trusted_binary_rejects_missing_helpers() { + let err = + find_trusted_binary("nsenter", &["relative-nsenter", "/missing/nsenter"]).unwrap_err(); + + assert!(err.to_string().contains("trusted nsenter helper not found")); + } + #[test] #[ignore = "requires root privileges"] fn test_create_and_drop_namespace() { @@ -850,10 +922,7 @@ mod tests { // Verify namespace exists let ns_path = format!("/var/run/netns/{name}"); - assert!( - std::path::Path::new(&ns_path).exists(), - "Namespace file should exist" - ); + assert!(Path::new(&ns_path).exists(), "Namespace file should exist"); // Verify IPs are set correctly assert_eq!( @@ -870,7 +939,7 @@ mod tests { // Verify namespace is gone assert!( - !std::path::Path::new(&ns_path).exists(), + !Path::new(&ns_path).exists(), "Namespace should be cleaned up" ); } diff --git a/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs b/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs index 854134cbf4..f614640234 100644 --- a/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs +++ b/crates/openshell-sandbox/src/sandbox/linux/seccomp.rs @@ -7,16 +7,16 @@ //! //! 1. **Socket domain blocks** -- prevent raw/kernel sockets that bypass the proxy //! 2. **Unconditional syscall blocks** -- block syscalls that enable sandbox escape -//! (fileless exec, ptrace, BPF, cross-process memory access, io_uring, mount) +//! (fileless exec, ptrace, BPF, cross-process memory access, `io_uring`, mount) //! 3. **Conditional syscall blocks** -- block dangerous flag combinations on otherwise -//! needed syscalls (execveat+AT_EMPTY_PATH, unshare+CLONE_NEWUSER, -//! seccomp+SET_MODE_FILTER) +//! needed syscalls (`execveat+AT_EMPTY_PATH`, `unshare+CLONE_NEWUSER`, +//! `seccomp+SET_MODE_FILTER`) use crate::policy::{NetworkMode, SandboxPolicy}; use miette::{IntoDiagnostic, Result}; use seccompiler::{ SeccompAction, SeccompCmpArgLen, SeccompCmpOp, SeccompCondition, SeccompFilter, SeccompRule, - apply_filter, + apply_filter, apply_filter_all_threads, }; use std::collections::BTreeMap; use std::convert::TryInto; @@ -25,12 +25,73 @@ use tracing::debug; /// Value of `SECCOMP_SET_MODE_FILTER` (linux/seccomp.h). const SECCOMP_SET_MODE_FILTER: u64 = 1; +/// Apply the supervisor seccomp filter across the running process. +/// +/// This runs after privileged startup helpers complete and synchronizes the +/// filter across all supervisor threads via TSYNC. It intentionally blocks +/// only the privileged escape primitives that the long-lived supervisor no +/// longer needs once bootstrap is complete. +pub fn apply_supervisor_prelude() -> Result<()> { + let filter = build_supervisor_prelude_filter()?; + set_no_new_privs()?; + apply_filter_all_threads(&filter).into_diagnostic()?; + Ok(()) +} + pub fn apply(policy: &SandboxPolicy) -> Result<()> { let allow_inet = matches!(policy.network.mode, NetworkMode::Proxy | NetworkMode::Allow); let main_filter = build_filter(allow_inet)?; let clone3_filter = build_clone3_filter()?; - // Required before applying seccomp filters. + set_no_new_privs()?; + apply_runtime_filters(&main_filter, &clone3_filter)?; + + Ok(()) +} + +fn build_filter(allow_inet: bool) -> Result { + let rules = build_filter_rules(allow_inet)?; + compile_filter(rules, SeccompAction::Errno(libc::EPERM as u32)) +} + +fn build_supervisor_prelude_filter() -> Result { + compile_filter( + build_supervisor_prelude_rules(), + SeccompAction::Errno(libc::EPERM as u32), + ) +} + +fn build_supervisor_prelude_rules() -> BTreeMap> { + let mut rules: BTreeMap> = BTreeMap::new(); + + for syscall in [ + libc::SYS_mount, + libc::SYS_fsopen, + libc::SYS_fsconfig, + libc::SYS_fsmount, + libc::SYS_fspick, + libc::SYS_move_mount, + libc::SYS_open_tree, + libc::SYS_pivot_root, + libc::SYS_umount2, + libc::SYS_bpf, + libc::SYS_perf_event_open, + libc::SYS_userfaultfd, + libc::SYS_init_module, + libc::SYS_finit_module, + libc::SYS_delete_module, + libc::SYS_kexec_load, + libc::SYS_kexec_file_load, + ] { + rules.entry(syscall).or_default(); + } + + rules +} + +fn set_no_new_privs() -> Result<()> { + // libc/syscall FFI requires unsafe + #[allow(unsafe_code)] let rc = unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) }; if rc != 0 { return Err(miette::miette!( @@ -39,25 +100,19 @@ pub fn apply(policy: &SandboxPolicy) -> Result<()> { )); } - apply_runtime_filters(&main_filter, &clone3_filter)?; - Ok(()) } -fn build_filter(allow_inet: bool) -> Result { - let rules = build_filter_rules(allow_inet)?; - +fn compile_filter( + rules: BTreeMap>, + blocked_action: SeccompAction, +) -> Result { let arch = std::env::consts::ARCH .try_into() .map_err(|_| miette::miette!("Unsupported architecture for seccomp"))?; - let filter = SeccompFilter::new( - rules, - SeccompAction::Allow, - SeccompAction::Errno(libc::EPERM as u32), - arch, - ) - .into_diagnostic()?; + let filter = + SeccompFilter::new(rules, SeccompAction::Allow, blocked_action, arch).into_diagnostic()?; filter.try_into().into_diagnostic() } @@ -66,7 +121,7 @@ fn build_filter(allow_inet: bool) -> Result { /// /// This is a separate filter from the main one because seccomp BPF cannot /// dereference the `struct clone_args *` pointer that clone3 takes as arg 0, -/// so we cannot selectively block CLONE_NEWUSER. We block clone3 +/// so we cannot selectively block `CLONE_NEWUSER`. We block clone3 /// unconditionally with ENOSYS so glibc falls back to the older clone /// syscall (where flags are a direct register argument and CAN be filtered). /// @@ -76,20 +131,7 @@ fn build_filter(allow_inet: bool) -> Result { fn build_clone3_filter() -> Result { let mut rules: BTreeMap> = BTreeMap::new(); rules.entry(libc::SYS_clone3).or_default(); - - let arch = std::env::consts::ARCH - .try_into() - .map_err(|_| miette::miette!("Unsupported architecture for seccomp"))?; - - let filter = SeccompFilter::new( - rules, - SeccompAction::Allow, - SeccompAction::Errno(libc::ENOSYS as u32), - arch, - ) - .into_diagnostic()?; - - filter.try_into().into_diagnostic() + compile_filter(rules, SeccompAction::Errno(libc::ENOSYS as u32)) } /// Install the sandbox seccomp filters in the required order. @@ -100,8 +142,8 @@ fn build_clone3_filter() -> Result { /// 2. Install the main filter second. It blocks further seccomp filter /// installation with `EPERM`, preserving the original hardening intent. fn apply_runtime_filters( - main_filter: seccompiler::BpfProgramRef, - clone3_filter: seccompiler::BpfProgramRef, + main_filter: seccompiler::BpfProgramRef<'_>, + clone3_filter: seccompiler::BpfProgramRef<'_>, ) -> Result<()> { apply_filter(clone3_filter).into_diagnostic()?; apply_filter(main_filter).into_diagnostic()?; @@ -112,11 +154,15 @@ fn build_filter_rules(allow_inet: bool) -> Result let mut rules: BTreeMap> = BTreeMap::new(); // --- Socket domain blocks --- - let mut blocked_domains = vec![libc::AF_PACKET, libc::AF_BLUETOOTH, libc::AF_VSOCK]; + let mut blocked_domains = vec![ + libc::AF_PACKET, + libc::AF_BLUETOOTH, + libc::AF_VSOCK, + libc::AF_NETLINK, + ]; if !allow_inet { blocked_domains.push(libc::AF_INET); blocked_domains.push(libc::AF_INET6); - blocked_domains.push(libc::AF_NETLINK); } for domain in blocked_domains { @@ -239,6 +285,15 @@ fn add_masked_arg_rule( } #[cfg(test)] +// libc/syscall FFI requires unsafe; these tests fork children and exercise +// blocked syscalls, so unsafe blocks/calls are pervasive. +#[allow( + unsafe_code, + unsafe_op_in_unsafe_fn, + unused_unsafe, + clippy::borrow_as_ptr, + trivial_numeric_casts +)] mod tests { use super::*; @@ -257,6 +312,15 @@ mod tests { assert!(filter.is_ok(), "build_filter(false) should succeed"); } + #[test] + fn build_supervisor_prelude_filter_compiles() { + let filter = build_supervisor_prelude_filter(); + assert!( + filter.is_ok(), + "build_supervisor_prelude_filter() should succeed" + ); + } + #[test] fn add_masked_arg_rule_creates_entry() { let mut rules: BTreeMap> = BTreeMap::new(); @@ -338,6 +402,57 @@ mod tests { } } + #[test] + fn supervisor_prelude_blocks_expected_syscalls() { + let filter_rules = build_supervisor_prelude_rules(); + + for syscall in [ + libc::SYS_mount, + libc::SYS_fsopen, + libc::SYS_fsconfig, + libc::SYS_fsmount, + libc::SYS_fspick, + libc::SYS_move_mount, + libc::SYS_open_tree, + libc::SYS_pivot_root, + libc::SYS_umount2, + libc::SYS_bpf, + libc::SYS_perf_event_open, + libc::SYS_userfaultfd, + libc::SYS_init_module, + libc::SYS_finit_module, + libc::SYS_delete_module, + libc::SYS_kexec_load, + libc::SYS_kexec_file_load, + ] { + assert!( + filter_rules.contains_key(&syscall), + "syscall {syscall} should be in the supervisor prelude rules" + ); + assert!( + filter_rules[&syscall].is_empty(), + "syscall {syscall} should be unconditionally blocked in the supervisor prelude" + ); + } + } + + #[test] + fn supervisor_prelude_keeps_required_setup_syscalls_available() { + let filter_rules = build_supervisor_prelude_rules(); + + for syscall in [ + libc::SYS_setns, + libc::SYS_clone, + libc::SYS_unshare, + libc::SYS_ptrace, + ] { + assert!( + !filter_rules.contains_key(&syscall), + "syscall {syscall} should remain available during supervisor startup" + ); + } + } + #[test] fn clone3_filter_compiles_and_blocks_clone3() { let bpf = build_clone3_filter(); @@ -444,6 +559,46 @@ mod tests { unsafe { assert_blocked_in_child(&filter, libc::SYS_setns, libc::EPERM) }; } + #[test] + fn behavioral_supervisor_prelude_mount_blocked() { + let pid = unsafe { libc::fork() }; + assert!(pid >= 0, "fork failed"); + if pid == 0 { + unsafe { + if let Err(err) = apply_supervisor_prelude() { + let msg = format!("failed to install supervisor prelude: {err}\n"); + libc::write(2, msg.as_ptr().cast(), msg.len()); + libc::_exit(1); + } + let ret = libc::syscall( + libc::SYS_mount, + std::ptr::null::(), + std::ptr::null::(), + std::ptr::null::(), + 0 as libc::c_ulong, + std::ptr::null::(), + ); + let errno = *libc::__errno_location(); + if ret == -1 && errno == libc::EPERM { + libc::_exit(0); + } else { + let msg = format!( + "mount: expected EPERM after supervisor prelude, got ret={ret} errno={errno}\n" + ); + libc::write(2, msg.as_ptr().cast(), msg.len()); + libc::_exit(1); + } + } + } + + let mut status: libc::c_int = 0; + unsafe { libc::waitpid(pid, &mut status, 0) }; + assert!( + unsafe { libc::WIFEXITED(status) && libc::WEXITSTATUS(status) == 0 }, + "mount should be blocked by the supervisor prelude filter" + ); + } + #[test] fn behavioral_clone3_returns_enosys() { // clone3 uses a separate filter that returns ENOSYS (not EPERM) so diff --git a/crates/openshell-sandbox/src/sandbox/mod.rs b/crates/openshell-sandbox/src/sandbox/mod.rs index f7b0373383..95aeae492f 100644 --- a/crates/openshell-sandbox/src/sandbox/mod.rs +++ b/crates/openshell-sandbox/src/sandbox/mod.rs @@ -14,6 +14,9 @@ pub mod linux; /// # Errors /// /// Returns an error if the sandbox cannot be applied. +// On Linux the spawn path uses `prepare`+`enforce` directly; this single-phase +// apply is only invoked from the non-Linux spawn_impl. +#[cfg_attr(target_os = "linux", allow(dead_code))] #[cfg_attr(not(target_os = "linux"), allow(clippy::unnecessary_wraps))] pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> { #[cfg(target_os = "linux")] @@ -38,3 +41,17 @@ pub fn apply(policy: &SandboxPolicy, workdir: Option<&str>) -> Result<()> { Ok(()) } } + +/// Apply seccomp hardening for the long-lived supervisor process itself. +#[cfg_attr(not(target_os = "linux"), allow(clippy::unnecessary_wraps))] +pub fn apply_supervisor_startup_hardening() -> Result<()> { + #[cfg(target_os = "linux")] + { + linux::apply_supervisor_prelude() + } + + #[cfg(not(target_os = "linux"))] + { + Ok(()) + } +} diff --git a/crates/openshell-sandbox/src/secrets.rs b/crates/openshell-sandbox/src/secrets.rs index a27537c910..63e253e50e 100644 --- a/crates/openshell-sandbox/src/secrets.rs +++ b/crates/openshell-sandbox/src/secrets.rs @@ -8,7 +8,7 @@ use std::fmt; const PLACEHOLDER_PREFIX: &str = "openshell:resolve:env:"; /// Public access to the placeholder prefix for fail-closed scanning in other modules. -pub(crate) const PLACEHOLDER_PREFIX_PUBLIC: &str = PLACEHOLDER_PREFIX; +pub const PLACEHOLDER_PREFIX_PUBLIC: &str = PLACEHOLDER_PREFIX; /// Characters that are valid in an env var key name (used to extract /// placeholder boundaries within concatenated strings like path segments). @@ -23,7 +23,7 @@ fn is_env_key_char(b: u8) -> bool { /// Error returned when a placeholder cannot be resolved or a resolved secret /// contains prohibited characters. #[derive(Debug)] -pub(crate) struct UnresolvedPlaceholderError { +pub struct UnresolvedPlaceholderError { pub location: &'static str, // "header", "query_param", "path" } @@ -39,18 +39,21 @@ impl fmt::Display for UnresolvedPlaceholderError { /// Result of rewriting an HTTP header block with credential resolution. #[derive(Debug)] -pub(crate) struct RewriteResult { +pub struct RewriteResult { /// The rewritten HTTP bytes (headers + body overflow). pub rewritten: Vec, /// A redacted version of the request target for logging. /// Contains `[CREDENTIAL]` in place of resolved credential values. /// `None` if the target was not modified. + // Kept on the public result struct as part of the API contract; consumed + // selectively by callers that emit redacted logs. + #[allow(dead_code)] pub redacted_target: Option, } /// Result of rewriting a request target for OPA evaluation. #[derive(Debug)] -pub(crate) struct RewriteTargetResult { +pub struct RewriteTargetResult { /// The resolved target (real secrets) — for upstream forwarding only. pub resolved: String, /// The redacted target (`[CREDENTIAL]` in place of secrets) — for OPA + logs. @@ -119,10 +122,9 @@ impl SecretResolver { .strip_prefix("Basic ") .or_else(|| trimmed.strip_prefix("basic ")) .map(str::trim) + && let Some(rewritten) = self.rewrite_basic_auth_token(encoded) { - if let Some(rewritten) = self.rewrite_basic_auth_token(encoded) { - return Some(format!("Basic {rewritten}")); - } + return Some(format!("Basic {rewritten}")); } // Prefixed placeholder: `Bearer openshell:resolve:env:KEY` @@ -172,7 +174,7 @@ impl SecretResolver { } } -pub(crate) fn placeholder_for_env_key(key: &str) -> String { +pub fn placeholder_for_env_key(key: &str) -> String { format!("{PLACEHOLDER_PREFIX}{key}") } @@ -226,12 +228,26 @@ fn percent_encode_path_segment(input: &str) -> String { let mut encoded = String::with_capacity(input.len()); for byte in input.bytes() { match byte { - // unreserved - b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'.' | b'_' | b'~' => { - encoded.push(byte as char); - } - // sub-delims + ":" + "@" - b'!' | b'$' | b'&' | b'\'' | b'(' | b')' | b'*' | b'+' | b',' | b';' | b'=' | b':' + // unreserved + sub-delims + ":" + "@" + b'A'..=b'Z' + | b'a'..=b'z' + | b'0'..=b'9' + | b'-' + | b'.' + | b'_' + | b'~' + | b'!' + | b'$' + | b'&' + | b'\'' + | b'(' + | b')' + | b'*' + | b'+' + | b',' + | b';' + | b'=' + | b':' | b'@' => { encoded.push(byte as char); } @@ -254,11 +270,11 @@ fn percent_decode(input: &str) -> String { let lo = bytes.next(); if let (Some(h), Some(l)) = (hi, lo) { let hex = [h, l]; - if let Ok(s) = std::str::from_utf8(&hex) { - if let Ok(val) = u8::from_str_radix(s, 16) { - decoded.push(val); - continue; - } + if let Ok(s) = std::str::from_utf8(&hex) + && let Ok(val) = u8::from_str_radix(s, 16) + { + decoded.push(val); + continue; } // Invalid percent encoding — preserve verbatim decoded.push(b'%'); @@ -318,46 +334,34 @@ struct RewriteLineResult { /// /// Given a request line like `GET /bot{TOKEN}/path?key={APIKEY} HTTP/1.1`, /// resolves placeholders in both path segments and query parameter values. +// `resolver` (the credential resolver) and `resolved` (the resolved string +// output) are intentionally distinct nouns; renaming would obscure intent. +#[allow(clippy::similar_names)] fn rewrite_request_line( line: &str, resolver: &SecretResolver, ) -> Result { // Request line format: METHOD SP REQUEST-URI SP HTTP-VERSION let mut parts = line.splitn(3, ' '); - let method = match parts.next() { - Some(m) => m, - None => { - return Ok(RewriteLineResult { - line: line.to_string(), - redacted_target: None, - }); - } + let unchanged = || { + Ok(RewriteLineResult { + line: line.to_string(), + redacted_target: None, + }) }; - let uri = match parts.next() { - Some(u) => u, - None => { - return Ok(RewriteLineResult { - line: line.to_string(), - redacted_target: None, - }); - } + let Some(method) = parts.next() else { + return unchanged(); }; - let version = match parts.next() { - Some(v) => v, - None => { - return Ok(RewriteLineResult { - line: line.to_string(), - redacted_target: None, - }); - } + let Some(uri) = parts.next() else { + return unchanged(); + }; + let Some(version) = parts.next() else { + return unchanged(); }; // Only rewrite if the URI contains a placeholder if !uri.contains(PLACEHOLDER_PREFIX) { - return Ok(RewriteLineResult { - line: line.to_string(), - redacted_target: None, - }); + return unchanged(); } // Split URI into path and query @@ -382,9 +386,10 @@ fn rewrite_request_line( }; // Reassemble - let resolved_uri = match &resolved_query { - Some(q) => format!("{resolved_path}?{q}"), - None => resolved_path.clone(), + let resolved_uri = if let Some(q) = resolved_query.as_ref() { + format!("{resolved_path}?{q}") + } else { + resolved_path }; let redacted_uri = match &redacted_query { Some(q) => format!("{redacted_path}?{q}"), @@ -404,6 +409,9 @@ fn rewrite_request_line( /// /// Returns `Some((resolved_path, redacted_path))` if any placeholders were found, /// `None` if no placeholders exist in the path. +// `resolver` and `resolved` are intentionally distinct nouns; see comment at +// `rewrite_request_line`. +#[allow(clippy::similar_names)] fn rewrite_uri_path( path: &str, resolver: &SecretResolver, @@ -446,6 +454,9 @@ fn rewrite_uri_path( /// /// Uses the placeholder grammar `openshell:resolve:env:[A-Za-z_][A-Za-z0-9_]*` /// to determine placeholder boundaries within concatenated text. +// `resolver` and `resolved` are intentionally distinct nouns; see comment at +// `rewrite_request_line`. +#[allow(clippy::similar_names)] fn rewrite_path_segment( segment: &str, resolver: &SecretResolver, @@ -561,7 +572,7 @@ fn rewrite_uri_query_params( /// /// Returns `Err` if any placeholder is detected but cannot be resolved /// (fail-closed behavior). -pub(crate) fn rewrite_http_header_block( +pub fn rewrite_http_header_block( raw: &[u8], resolver: Option<&SecretResolver>, ) -> Result { @@ -627,22 +638,25 @@ pub(crate) fn rewrite_http_header_block( }) } -pub(crate) fn rewrite_header_line(line: &str, resolver: &SecretResolver) -> String { +pub fn rewrite_header_line(line: &str, resolver: &SecretResolver) -> String { let Some((name, value)) = line.split_once(':') else { return line.to_string(); }; - match resolver.rewrite_header_value(value.trim()) { - Some(rewritten) => format!("{name}: {rewritten}"), - None => line.to_string(), - } + resolver.rewrite_header_value(value.trim()).map_or_else( + || line.to_string(), + |rewritten| format!("{name}: {rewritten}"), + ) } /// Resolve placeholders in a request target (path + query) for OPA evaluation. /// /// Returns the resolved target (real secrets, for upstream) and a redacted /// version (`[CREDENTIAL]` in place of secrets, for OPA input and logs). -pub(crate) fn rewrite_target_for_eval( +// `resolver` and `resolved` are intentionally distinct nouns; see comment at +// `rewrite_request_line`. +#[allow(clippy::similar_names)] +pub fn rewrite_target_for_eval( target: &str, resolver: &SecretResolver, ) -> Result { @@ -695,6 +709,10 @@ pub(crate) fn rewrite_target_for_eval( // --------------------------------------------------------------------------- #[cfg(test)] +#[allow( + clippy::iter_on_single_items, + reason = "Test code: single-key fixtures are clearer as array literals than std::iter::once." +)] mod tests { use super::*; diff --git a/crates/openshell-sandbox/src/ssh.rs b/crates/openshell-sandbox/src/ssh.rs index 3c26f80611..9434d0a16b 100644 --- a/crates/openshell-sandbox/src/ssh.rs +++ b/crates/openshell-sandbox/src/ssh.rs @@ -13,8 +13,7 @@ use miette::{IntoDiagnostic, Result}; use nix::pty::{Winsize, openpty}; use nix::unistd::setsid; use openshell_ocsf::{ - ActionId, ActivityId, AuthTypeId, ConfidenceId, DetectionFindingBuilder, DispositionId, - FindingInfo, SeverityId, SshActivityBuilder, StatusId, ocsf_emit, + ActionId, ActivityId, DispositionId, SeverityId, SshActivityBuilder, StatusId, ocsf_emit, }; use rand_core::OsRng; use russh::keys::{Algorithm, PrivateKey}; @@ -22,36 +21,27 @@ use russh::server::{Auth, Handle, Session}; use russh::{ChannelId, CryptoVec}; use std::collections::HashMap; use std::io::{Read, Write}; -use std::net::SocketAddr; use std::os::fd::{AsRawFd, RawFd}; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; -use std::sync::{Arc, Mutex, mpsc}; -use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; -use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use tokio::net::TcpListener; +use std::sync::{Arc, mpsc}; +use std::time::Duration; +use tokio::net::UnixListener; use tracing::warn; -const PREFACE_MAGIC: &str = "NSSH1"; -#[cfg(test)] -const SSH_HANDSHAKE_SECRET_ENV: &str = "OPENSHELL_SSH_HANDSHAKE_SECRET"; - -/// A time-bounded set of nonces used to detect replayed NSSH1 handshakes. -/// Each entry records the `Instant` it was inserted; a background reaper task -/// periodically evicts entries older than the handshake skew window. -type NonceCache = Arc>>; - /// Perform SSH server initialization: generate a host key, build the config, -/// and bind the TCP listener. Extracted so that startup errors can be forwarded -/// through the readiness channel rather than being silently logged. -async fn ssh_server_init( - listen_addr: SocketAddr, - ca_file_paths: &Option<(PathBuf, PathBuf)>, -) -> Result<( - TcpListener, +/// and bind the Unix socket listener. Extracted so that startup errors can be +/// forwarded through the readiness channel rather than being silently logged. +type SshServerInit = ( + UnixListener, Arc, Option>, -)> { +); + +fn ssh_server_init( + listen_path: &Path, + ca_file_paths: &Option<(PathBuf, PathBuf)>, +) -> Result { let mut rng = OsRng; let host_key = PrivateKey::random(&mut rng, Algorithm::Ed25519).into_diagnostic()?; @@ -63,14 +53,43 @@ async fn ssh_server_init( let config = Arc::new(config); let ca_paths = ca_file_paths.as_ref().map(|p| Arc::new(p.clone())); - let listener = TcpListener::bind(listen_addr).await.into_diagnostic()?; + + // Ensure the parent directory exists and is root-owned with 0700 + // permissions. The sandbox entrypoint runs as an unprivileged user; it + // must not be able to enter this directory and connect to the socket. + if let Some(parent) = listen_path.parent() { + std::fs::create_dir_all(parent).into_diagnostic()?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let perms = std::fs::Permissions::from_mode(0o700); + std::fs::set_permissions(parent, perms).into_diagnostic()?; + } + } + + // Remove any stale socket from a previous run before binding. + if listen_path.exists() { + std::fs::remove_file(listen_path).into_diagnostic()?; + } + let listener = UnixListener::bind(listen_path).into_diagnostic()?; + + // Tighten permissions so only the supervisor (root) can connect. The + // sandbox entrypoint runs as an unprivileged user and must not be able to + // dial the SSH daemon directly — all access goes through the relay from + // the gateway. + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let perms = std::fs::Permissions::from_mode(0o600); + std::fs::set_permissions(listen_path, perms).into_diagnostic()?; + } + ocsf_emit!( SshActivityBuilder::new(crate::ocsf_ctx()) .activity(ActivityId::Listen) .severity(SeverityId::Informational) .status(StatusId::Success) - .src_endpoint_addr(listen_addr.ip(), listen_addr.port()) - .message(format!("SSH server listening on {listen_addr}")) + .message(format!("SSH server listening on {}", listen_path.display())) .build() ); @@ -79,18 +98,16 @@ async fn ssh_server_init( #[allow(clippy::too_many_arguments)] pub async fn run_ssh_server( - listen_addr: SocketAddr, + listen_path: PathBuf, ready_tx: tokio::sync::oneshot::Sender>, policy: SandboxPolicy, workdir: Option, - handshake_secret: String, - handshake_skew_secs: u64, netns_fd: Option, proxy_url: Option, ca_file_paths: Option<(PathBuf, PathBuf)>, provider_env: HashMap, ) -> Result<()> { - let (listener, config, ca_paths) = match ssh_server_init(listen_addr, &ca_file_paths).await { + let (listener, config, ca_paths) = match ssh_server_init(&listen_path, &ca_file_paths) { Ok(v) => { // Signal that the SSH server has bound the socket and is ready to // accept connections. The parent task awaits this before spawning @@ -105,49 +122,25 @@ pub async fn run_ssh_server( } }; - // Nonce cache for replay detection. Entries are evicted by a background - // reaper once they exceed the handshake skew window. - let nonce_cache: NonceCache = Arc::new(Mutex::new(HashMap::new())); - - // Background task that periodically purges expired nonces. - let reaper_cache = nonce_cache.clone(); - let ttl = Duration::from_secs(handshake_skew_secs); - tokio::spawn(async move { - let mut interval = tokio::time::interval(Duration::from_secs(60)); - loop { - interval.tick().await; - if let Ok(mut cache) = reaper_cache.lock() { - cache.retain(|_, inserted| inserted.elapsed() < ttl); - } - } - }); - loop { - let (stream, peer) = listener.accept().await.into_diagnostic()?; - stream.set_nodelay(true).into_diagnostic()?; + let (stream, _peer) = listener.accept().await.into_diagnostic()?; let config = config.clone(); let policy = policy.clone(); let workdir = workdir.clone(); - let secret = handshake_secret.clone(); let proxy_url = proxy_url.clone(); let ca_paths = ca_paths.clone(); let provider_env = provider_env.clone(); - let nonce_cache = nonce_cache.clone(); tokio::spawn(async move { if let Err(err) = handle_connection( stream, - peer, config, policy, workdir, - &secret, - handshake_skew_secs, netns_fd, proxy_url, ca_paths, provider_env, - &nonce_cache, ) .await { @@ -166,41 +159,18 @@ pub async fn run_ssh_server( #[allow(clippy::too_many_arguments)] async fn handle_connection( - mut stream: tokio::net::TcpStream, - peer: SocketAddr, + stream: tokio::net::UnixStream, config: Arc, policy: SandboxPolicy, workdir: Option, - secret: &str, - handshake_skew_secs: u64, netns_fd: Option, proxy_url: Option, ca_file_paths: Option>, provider_env: HashMap, - nonce_cache: &NonceCache, ) -> Result<()> { - tracing::debug!(peer = %peer, "SSH connection: reading handshake preface"); - let mut line = String::new(); - read_line(&mut stream, &mut line).await?; - tracing::debug!(peer = %peer, preface_len = line.len(), "SSH connection: preface received, verifying"); - if !verify_preface(&line, secret, handshake_skew_secs, nonce_cache)? { - ocsf_emit!( - SshActivityBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .src_endpoint_addr(peer.ip(), peer.port()) - .message(format!( - "SSH connection: handshake verification failed from {peer}" - )) - .build() - ); - let _ = stream.write_all(b"ERR\n").await; - return Ok(()); - } - stream.write_all(b"OK\n").await.into_diagnostic()?; + // Access is gated by the Unix-socket filesystem permissions (root-only), + // not by an application-level preface. The supervisor bridges the + // gateway's RelayStream directly into this socket. ocsf_emit!( SshActivityBuilder::new(crate::ocsf_ctx()) .activity(ActivityId::Open) @@ -208,9 +178,7 @@ async fn handle_connection( .disposition(DispositionId::Allowed) .severity(SeverityId::Informational) .status(StatusId::Success) - .src_endpoint_addr(peer.ip(), peer.port()) - .auth_type(AuthTypeId::Other, "NSSH1") - .message(format!("SSH handshake accepted from {peer}")) + .message("SSH connection accepted on supervisor Unix socket") .build() ); @@ -228,107 +196,6 @@ async fn handle_connection( Ok(()) } -async fn read_line(stream: &mut tokio::net::TcpStream, buf: &mut String) -> Result<()> { - let mut bytes = Vec::new(); - loop { - let mut byte = [0u8; 1]; - let n = stream.read(&mut byte).await.into_diagnostic()?; - if n == 0 { - break; - } - if byte[0] == b'\n' { - break; - } - bytes.push(byte[0]); - if bytes.len() > 1024 { - break; - } - } - *buf = String::from_utf8_lossy(&bytes).to_string(); - Ok(()) -} - -fn verify_preface( - line: &str, - secret: &str, - handshake_skew_secs: u64, - nonce_cache: &NonceCache, -) -> Result { - let parts: Vec<&str> = line.split_whitespace().collect(); - if parts.len() != 5 || parts[0] != PREFACE_MAGIC { - return Ok(false); - } - let token = parts[1]; - let timestamp: i64 = parts[2].parse().unwrap_or(0); - let nonce = parts[3]; - let signature = parts[4]; - - let now = i64::try_from( - SystemTime::now() - .duration_since(UNIX_EPOCH) - .into_diagnostic()? - .as_secs(), - ) - .into_diagnostic()?; - let skew = (now - timestamp).unsigned_abs(); - if skew > handshake_skew_secs { - return Ok(false); - } - - let payload = format!("{token}|{timestamp}|{nonce}"); - let expected = hmac_sha256(secret.as_bytes(), payload.as_bytes()); - if signature != expected { - return Ok(false); - } - - // Reject replayed nonces. The cache is bounded by the reaper task which - // evicts entries older than `handshake_skew_secs`. - let mut cache = nonce_cache - .lock() - .map_err(|_| miette::miette!("nonce cache lock poisoned"))?; - if cache.contains_key(nonce) { - ocsf_emit!( - SshActivityBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Other) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::High) - .auth_type(AuthTypeId::Other, "NSSH1") - .message(format!("NSSH1 nonce replay detected: {nonce}")) - .build() - ); - ocsf_emit!( - DetectionFindingBuilder::new(crate::ocsf_ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::High) - .is_alert(true) - .confidence(ConfidenceId::High) - .finding_info(FindingInfo::new( - "nssh1-nonce-replay", - "NSSH1 Nonce Replay Attack" - )) - .evidence("nonce", nonce) - .build() - ); - return Ok(false); - } - cache.insert(nonce.to_string(), Instant::now()); - - Ok(true) -} - -fn hmac_sha256(key: &[u8], data: &[u8]) -> String { - use hmac::{Hmac, Mac}; - use sha2::Sha256; - - let mut mac = Hmac::::new_from_slice(key).expect("hmac key"); - mac.update(data); - let result = mac.finalize().into_bytes(); - hex::encode(result) -} - /// Per-channel state for tracking PTY resources and I/O senders. /// /// Each SSH channel gets its own PTY master (if a PTY was requested) and input @@ -400,8 +267,8 @@ impl russh::server::Handler for SshHandler { /// Clean up per-channel state when the channel is closed. /// /// This is the final cleanup and subsumes `channel_eof` — if `channel_close` - /// fires without a preceding `channel_eof`, all resources (pty_master File, - /// input_sender) are dropped here. + /// fires without a preceding `channel_eof`, all resources (`pty_master` File, + /// `input_sender`) are dropped here. async fn channel_close( &mut self, channel: ChannelId, @@ -452,7 +319,9 @@ impl russh::server::Handler for SshHandler { } let host = host_to_connect.to_string(); - let port = port_to_connect as u16; + // SSH protocol port is bounded by u32 but only u16 is meaningful; + // saturate as a guard for malformed clients. + let port = u16::try_from(port_to_connect).unwrap_or(u16::MAX); let netns_fd = self.netns_fd; tokio::spawn(async move { @@ -787,8 +656,10 @@ fn session_user_and_home(policy: &SandboxPolicy) -> (String, String) { let home = nix::unistd::User::from_name(user) .ok() .flatten() - .map(|u| u.dir.to_string_lossy().into_owned()) - .unwrap_or_else(|| format!("/home/{user}")); + .map_or_else( + || format!("/home/{user}"), + |u| u.dir.to_string_lossy().into_owned(), + ); (user.to_string(), home) } _ => ("sandbox".to_string(), "/sandbox".to_string()), @@ -1174,6 +1045,9 @@ mod unsafe_pty { } #[allow(unsafe_code)] + // `libc::TIOCSCTTY` is `u32` on macOS/BSD and `u64` on Linux; allow the + // cross-platform conversion so the same expression compiles everywhere. + #[allow(clippy::useless_conversion)] fn set_controlling_tty(fd: RawFd) -> std::io::Result<()> { let rc = unsafe { libc::ioctl(fd, libc::TIOCSCTTY.into(), 0) }; if rc != 0 { @@ -1317,6 +1191,11 @@ fn is_loopback_host(host: &str) -> bool { } #[cfg(test)] +#[allow( + clippy::doc_markdown, + unsafe_code, + reason = "Test code: doc text references identifiers and uses libc::winsize zero-init." +)] mod tests { use super::*; use std::process::Stdio; @@ -1424,136 +1303,6 @@ mod tests { ); } - // ----------------------------------------------------------------------- - // verify_preface tests - // ----------------------------------------------------------------------- - - /// Build a valid NSSH1 preface line with the given parameters. - fn build_preface(token: &str, secret: &str, nonce: &str, timestamp: i64) -> String { - let payload = format!("{token}|{timestamp}|{nonce}"); - let signature = hmac_sha256(secret.as_bytes(), payload.as_bytes()); - format!("{PREFACE_MAGIC} {token} {timestamp} {nonce} {signature}") - } - - fn fresh_nonce_cache() -> NonceCache { - Arc::new(Mutex::new(HashMap::new())) - } - - fn current_timestamp() -> i64 { - i64::try_from( - SystemTime::now() - .duration_since(UNIX_EPOCH) - .unwrap() - .as_secs(), - ) - .unwrap() - } - - #[test] - fn verify_preface_accepts_valid_preface() { - let secret = "test-secret-key"; - let nonce = "unique-nonce-1"; - let ts = current_timestamp(); - let line = build_preface("tok1", secret, nonce, ts); - let cache = fresh_nonce_cache(); - - assert!(verify_preface(&line, secret, 300, &cache).unwrap()); - } - - #[test] - fn verify_preface_rejects_replayed_nonce() { - let secret = "test-secret-key"; - let nonce = "replay-nonce"; - let ts = current_timestamp(); - let line = build_preface("tok1", secret, nonce, ts); - let cache = fresh_nonce_cache(); - - // First attempt should succeed. - assert!(verify_preface(&line, secret, 300, &cache).unwrap()); - // Second attempt with the same nonce should be rejected. - assert!(!verify_preface(&line, secret, 300, &cache).unwrap()); - } - - #[test] - fn verify_preface_rejects_expired_timestamp() { - let secret = "test-secret-key"; - let nonce = "expired-nonce"; - // Timestamp 600 seconds in the past, with a 300-second skew window. - let ts = current_timestamp() - 600; - let line = build_preface("tok1", secret, nonce, ts); - let cache = fresh_nonce_cache(); - - assert!(!verify_preface(&line, secret, 300, &cache).unwrap()); - } - - #[test] - fn verify_preface_rejects_invalid_hmac() { - let secret = "test-secret-key"; - let nonce = "hmac-nonce"; - let ts = current_timestamp(); - // Build with the correct secret, then verify with the wrong one. - let line = build_preface("tok1", secret, nonce, ts); - let cache = fresh_nonce_cache(); - - assert!(!verify_preface(&line, "wrong-secret", 300, &cache).unwrap()); - } - - #[test] - fn verify_preface_rejects_malformed_input() { - let cache = fresh_nonce_cache(); - - // Too few parts. - assert!(!verify_preface("NSSH1 tok1 123", "s", 300, &cache).unwrap()); - // Wrong magic. - assert!(!verify_preface("NSSH2 tok1 123 nonce sig", "s", 300, &cache).unwrap()); - // Empty string. - assert!(!verify_preface("", "s", 300, &cache).unwrap()); - } - - #[test] - fn verify_preface_distinct_nonces_both_accepted() { - let secret = "test-secret-key"; - let ts = current_timestamp(); - let cache = fresh_nonce_cache(); - - let line1 = build_preface("tok1", secret, "nonce-a", ts); - let line2 = build_preface("tok1", secret, "nonce-b", ts); - - assert!(verify_preface(&line1, secret, 300, &cache).unwrap()); - assert!(verify_preface(&line2, secret, 300, &cache).unwrap()); - } - - #[test] - fn apply_child_env_keeps_handshake_secret_out_of_ssh_children() { - let mut cmd = Command::new("/usr/bin/env"); - cmd.env(SSH_HANDSHAKE_SECRET_ENV, "should-not-leak") - .stdin(Stdio::null()) - .stdout(Stdio::piped()) - .stderr(Stdio::null()); - - let provider_env = std::iter::once(( - "ANTHROPIC_API_KEY".to_string(), - "openshell:resolve:env:ANTHROPIC_API_KEY".to_string(), - )) - .collect(); - - apply_child_env( - &mut cmd, - "/sandbox", - "sandbox", - "dumb", - None, - None, - &provider_env, - ); - - let output = cmd.output().expect("spawn env"); - let stdout = String::from_utf8(output.stdout).expect("utf8"); - - assert!(!stdout.contains(SSH_HANDSHAKE_SECRET_ENV)); - assert!(stdout.contains("ANTHROPIC_API_KEY=openshell:resolve:env:ANTHROPIC_API_KEY")); - } - // ----------------------------------------------------------------------- // SEC-007: is_loopback_host tests // ----------------------------------------------------------------------- @@ -1710,4 +1459,73 @@ mod tests { .unwrap(); assert_eq!(rx_b.recv().unwrap(), b"still-alive"); } + + /// `install_pre_exec_no_pty` runs drop_privileges and succeeds when the + /// current user/group is already the configured one (no actual uid change). + /// + /// This exercises the pre_exec hook end-to-end without needing root: a policy + /// with no run_as_user/group is a no-op when the process is already unprivileged. + #[cfg(unix)] + #[test] + fn pre_exec_always_calls_drop_privileges() { + use crate::policy::{ + FilesystemPolicy, LandlockPolicy, NetworkPolicy, ProcessPolicy, SandboxPolicy, + }; + + // No user/group configured and not running as root → drop_privileges is + // a no-op, so spawn succeeds regardless of the effective UID. + let policy = SandboxPolicy { + version: 0, + filesystem: FilesystemPolicy::default(), + network: NetworkPolicy::default(), + landlock: LandlockPolicy::default(), + process: ProcessPolicy { + run_as_user: None, + run_as_group: None, + }, + }; + + // Skip if running as root: drop_privileges would try to switch to + // "sandbox" which may not exist in the test environment. + if nix::unistd::geteuid().is_root() { + return; + } + + let mut cmd = Command::new("echo"); + cmd.arg("drop-privileges-ok"); + cmd.stdout(Stdio::piped()); + + unsafe_pty::install_pre_exec_no_pty( + &mut cmd, + policy, + None, + None, // no netns fd + #[cfg(target_os = "linux")] + sandbox::linux::prepare( + &SandboxPolicy { + version: 0, + filesystem: FilesystemPolicy::default(), + network: NetworkPolicy::default(), + landlock: LandlockPolicy::default(), + process: ProcessPolicy { + run_as_user: None, + run_as_group: None, + }, + }, + None, + ) + .expect("prepare should succeed in test environment"), + ); + + let output = cmd + .spawn() + .expect("spawn must succeed") + .wait_with_output() + .expect("wait_with_output"); + assert!(output.status.success(), "echo should exit 0"); + assert!( + String::from_utf8_lossy(&output.stdout).contains("drop-privileges-ok"), + "echo output should contain 'drop-privileges-ok'" + ); + } } diff --git a/crates/openshell-sandbox/src/supervisor_session.rs b/crates/openshell-sandbox/src/supervisor_session.rs new file mode 100644 index 0000000000..490a0cba77 --- /dev/null +++ b/crates/openshell-sandbox/src/supervisor_session.rs @@ -0,0 +1,570 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Persistent supervisor-to-gateway session. +//! +//! Maintains a long-lived `ConnectSupervisor` bidirectional gRPC stream to the +//! gateway. When the gateway sends `RelayOpen`, the supervisor initiates a +//! `RelayStream` gRPC call (a new HTTP/2 stream multiplexed over the same +//! TCP+TLS connection as the control stream) and bridges it to the local SSH +//! daemon. The supervisor is a dumb byte bridge — it has no protocol awareness +//! of the SSH or NSSH1 bytes flowing through. + +use std::time::Duration; + +use openshell_core::proto::open_shell_client::OpenShellClient; +use openshell_core::proto::{ + GatewayMessage, RelayFrame, RelayInit, SupervisorHeartbeat, SupervisorHello, SupervisorMessage, + gateway_message, supervisor_message, +}; +use openshell_ocsf::{ + ActivityId, Endpoint, NetworkActivityBuilder, OcsfEvent, SandboxContext, SeverityId, StatusId, + ocsf_emit, +}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::sync::mpsc; +use tokio_stream::StreamExt; +use tonic::transport::Channel; +use tracing::{debug, warn}; + +use crate::grpc_client; + +const INITIAL_BACKOFF: Duration = Duration::from_secs(1); +const MAX_BACKOFF: Duration = Duration::from_secs(30); + +/// Parse a gRPC endpoint URI into an OCSF `Endpoint` (host + port). Falls back +/// to treating the whole string as a domain if parsing fails. +fn ocsf_gateway_endpoint(endpoint: &str) -> Endpoint { + let without_scheme = endpoint + .split_once("://") + .map_or(endpoint, |(_, rest)| rest); + let host_and_port = without_scheme.split('/').next().unwrap_or(without_scheme); + if let Some((host, port)) = host_and_port.rsplit_once(':') + && let Ok(port) = port.parse::() + { + return Endpoint::from_domain(host, port); + } + Endpoint::from_domain(host_and_port, 0) +} + +fn session_established_event( + ctx: &SandboxContext, + endpoint: &str, + session_id: &str, + heartbeat_secs: u32, +) -> OcsfEvent { + NetworkActivityBuilder::new(ctx) + .activity(ActivityId::Open) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .dst_endpoint(ocsf_gateway_endpoint(endpoint)) + .message(format!( + "supervisor session established (session_id={session_id}, heartbeat_secs={heartbeat_secs})" + )) + .build() +} + +fn session_closed_event(ctx: &SandboxContext, endpoint: &str, sandbox_id: &str) -> OcsfEvent { + NetworkActivityBuilder::new(ctx) + .activity(ActivityId::Close) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .dst_endpoint(ocsf_gateway_endpoint(endpoint)) + .message(format!("supervisor session ended cleanly ({sandbox_id})")) + .build() +} + +fn session_failed_event( + ctx: &SandboxContext, + endpoint: &str, + attempt: u64, + error: &str, +) -> OcsfEvent { + NetworkActivityBuilder::new(ctx) + .activity(ActivityId::Fail) + .severity(SeverityId::Low) + .status(StatusId::Failure) + .dst_endpoint(ocsf_gateway_endpoint(endpoint)) + .message(format!( + "supervisor session failed, reconnecting (attempt {attempt}): {error}" + )) + .build() +} + +fn relay_open_event(ctx: &SandboxContext, channel_id: &str) -> OcsfEvent { + NetworkActivityBuilder::new(ctx) + .activity(ActivityId::Open) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .message(format!("relay open (channel_id={channel_id})")) + .build() +} + +fn relay_closed_event(ctx: &SandboxContext, channel_id: &str) -> OcsfEvent { + NetworkActivityBuilder::new(ctx) + .activity(ActivityId::Close) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .message(format!("relay closed (channel_id={channel_id})")) + .build() +} + +fn relay_failed_event(ctx: &SandboxContext, channel_id: &str, error: &str) -> OcsfEvent { + NetworkActivityBuilder::new(ctx) + .activity(ActivityId::Fail) + .severity(SeverityId::Low) + .status(StatusId::Failure) + .message(format!( + "relay bridge failed (channel_id={channel_id}): {error}" + )) + .build() +} + +fn relay_close_from_gateway_event( + ctx: &SandboxContext, + channel_id: &str, + reason: &str, +) -> OcsfEvent { + NetworkActivityBuilder::new(ctx) + .activity(ActivityId::Close) + .severity(SeverityId::Informational) + .message(format!( + "relay close from gateway (channel_id={channel_id}, reason={reason})" + )) + .build() +} + +/// Size of chunks read from the local SSH socket when forwarding bytes back +/// to the gateway over the gRPC response stream. 16 KiB matches the default +/// HTTP/2 frame size so each `RelayFrame::data` fits in one frame. +const RELAY_CHUNK_SIZE: usize = 16 * 1024; + +fn map_stream_message( + message: Result, tonic::Status>, + eof_error: &'static str, +) -> Result> { + match message { + Ok(Some(msg)) => Ok(msg), + Ok(None) => Err(eof_error.into()), + Err(e) => Err(format!("stream error: {e}").into()), + } +} + +/// Spawn the supervisor session task. +/// +/// The task runs for the lifetime of the sandbox process, reconnecting with +/// exponential backoff on failures. +pub fn spawn( + endpoint: String, + sandbox_id: String, + ssh_socket_path: std::path::PathBuf, +) -> tokio::task::JoinHandle<()> { + tokio::spawn(run_session_loop(endpoint, sandbox_id, ssh_socket_path)) +} + +async fn run_session_loop( + endpoint: String, + sandbox_id: String, + ssh_socket_path: std::path::PathBuf, +) { + let mut backoff = INITIAL_BACKOFF; + let mut attempt: u64 = 0; + + loop { + attempt += 1; + + match run_single_session(&endpoint, &sandbox_id, &ssh_socket_path).await { + Ok(()) => { + let event = session_closed_event(crate::ocsf_ctx(), &endpoint, &sandbox_id); + ocsf_emit!(event); + break; + } + Err(e) => { + let event = + session_failed_event(crate::ocsf_ctx(), &endpoint, attempt, &e.to_string()); + ocsf_emit!(event); + tokio::time::sleep(backoff).await; + backoff = (backoff * 2).min(MAX_BACKOFF); + } + } + } +} + +async fn run_single_session( + endpoint: &str, + sandbox_id: &str, + ssh_socket_path: &std::path::Path, +) -> Result<(), Box> { + // Connect to the gateway. The same `Channel` is used for both the + // long-lived control stream and all data-plane `RelayStream` calls, so + // every relay rides the same TCP+TLS+HTTP/2 connection — no new TLS + // handshake per relay. + let channel = grpc_client::connect_channel_pub(endpoint) + .await + .map_err(|e| format!("connect failed: {e}"))?; + let mut client = OpenShellClient::new(channel.clone()); + + // Create the outbound message stream. + let (tx, rx) = mpsc::channel::(64); + let outbound = tokio_stream::wrappers::ReceiverStream::new(rx); + + // Send hello as the first message. + let instance_id = uuid::Uuid::new_v4().to_string(); + tx.send(SupervisorMessage { + payload: Some(supervisor_message::Payload::Hello(SupervisorHello { + sandbox_id: sandbox_id.to_string(), + instance_id: instance_id.clone(), + })), + }) + .await + .map_err(|_| "failed to queue hello")?; + + // Open the bidirectional stream. + let response = client + .connect_supervisor(outbound) + .await + .map_err(|e| format!("connect_supervisor RPC failed: {e}"))?; + let mut inbound = response.into_inner(); + + // Wait for SessionAccepted. + let accepted = match map_stream_message( + inbound.message().await, + "stream closed before session accepted", + )? + .payload + { + Some(gateway_message::Payload::SessionAccepted(a)) => a, + Some(gateway_message::Payload::SessionRejected(r)) => { + return Err(format!("session rejected: {}", r.reason).into()); + } + _ => return Err("expected SessionAccepted or SessionRejected".into()), + }; + + let heartbeat_secs = accepted.heartbeat_interval_secs.max(5); + let event = session_established_event( + crate::ocsf_ctx(), + endpoint, + &accepted.session_id, + heartbeat_secs, + ); + ocsf_emit!(event); + + // Main loop: receive gateway messages + send heartbeats. + let mut heartbeat_interval = + tokio::time::interval(Duration::from_secs(u64::from(heartbeat_secs))); + heartbeat_interval.tick().await; // skip immediate tick + + loop { + tokio::select! { + msg = inbound.message() => { + let msg = map_stream_message(msg, "gateway closed stream")?; + handle_gateway_message( + &msg, + sandbox_id, + ssh_socket_path, + &channel, + ); + } + _ = heartbeat_interval.tick() => { + let hb = SupervisorMessage { + payload: Some(supervisor_message::Payload::Heartbeat( + SupervisorHeartbeat {}, + )), + }; + if tx.send(hb).await.is_err() { + return Err("outbound channel closed".into()); + } + } + } + } +} + +fn handle_gateway_message( + msg: &GatewayMessage, + sandbox_id: &str, + ssh_socket_path: &std::path::Path, + channel: &Channel, +) { + match &msg.payload { + Some(gateway_message::Payload::Heartbeat(_)) => { + // Gateway heartbeat — nothing to do. + } + Some(gateway_message::Payload::RelayOpen(open)) => { + let channel_id = open.channel_id.clone(); + let sandbox_id = sandbox_id.to_string(); + let channel = channel.clone(); + let ssh_socket_path = ssh_socket_path.to_path_buf(); + + let event = relay_open_event(crate::ocsf_ctx(), &channel_id); + ocsf_emit!(event); + + tokio::spawn(async move { + match handle_relay_open(&channel_id, &ssh_socket_path, channel).await { + Ok(()) => { + let event = relay_closed_event(crate::ocsf_ctx(), &channel_id); + ocsf_emit!(event); + } + Err(e) => { + let event = + relay_failed_event(crate::ocsf_ctx(), &channel_id, &e.to_string()); + ocsf_emit!(event); + warn!( + sandbox_id = %sandbox_id, + channel_id = %channel_id, + error = %e, + "supervisor session: relay bridge failed" + ); + } + } + }); + } + Some(gateway_message::Payload::RelayClose(close)) => { + let event = + relay_close_from_gateway_event(crate::ocsf_ctx(), &close.channel_id, &close.reason); + ocsf_emit!(event); + } + _ => { + warn!(sandbox_id = %sandbox_id, "supervisor session: unexpected gateway message"); + } + } +} + +/// Handle a `RelayOpen` by initiating a `RelayStream` RPC on the gateway and +/// bridging that stream to the local SSH daemon. +/// +/// This opens a new HTTP/2 stream on the existing `Channel` — no new TCP or +/// TLS handshake. The first `RelayFrame` we send is a `RelayInit`; subsequent +/// frames carry raw SSH bytes in `data`. +async fn handle_relay_open( + channel_id: &str, + ssh_socket_path: &std::path::Path, + channel: Channel, +) -> Result<(), Box> { + let mut client = OpenShellClient::new(channel); + + // Outbound chunks to the gateway. + let (out_tx, out_rx) = mpsc::channel::(16); + let outbound = tokio_stream::wrappers::ReceiverStream::new(out_rx); + + // First frame: identify the channel. + out_tx + .send(RelayFrame { + payload: Some(openshell_core::proto::relay_frame::Payload::Init( + RelayInit { + channel_id: channel_id.to_string(), + }, + )), + }) + .await + .map_err(|_| "outbound channel closed before init")?; + + // Initiate the RPC. This rides the existing HTTP/2 connection. + let response = client + .relay_stream(outbound) + .await + .map_err(|e| format!("relay_stream RPC failed: {e}"))?; + let mut inbound = response.into_inner(); + + // Connect to the local SSH daemon on its Unix socket. + let ssh = tokio::net::UnixStream::connect(ssh_socket_path).await?; + let (mut ssh_r, mut ssh_w) = ssh.into_split(); + + debug!( + channel_id = %channel_id, + socket = %ssh_socket_path.display(), + "relay bridge: connected to local SSH daemon" + ); + + // SSH → gRPC (out_tx): read local SSH, forward as `RelayFrame::data`. + let out_tx_writer = out_tx.clone(); + let ssh_to_grpc = tokio::spawn(async move { + let mut buf = vec![0u8; RELAY_CHUNK_SIZE]; + loop { + match ssh_r.read(&mut buf).await { + Ok(0) | Err(_) => break, + Ok(n) => { + let chunk = RelayFrame { + payload: Some(openshell_core::proto::relay_frame::Payload::Data( + buf[..n].to_vec(), + )), + }; + if out_tx_writer.send(chunk).await.is_err() { + break; + } + } + } + } + }); + + // gRPC (inbound) → SSH: drain inbound chunks into the local SSH socket. + let mut inbound_err: Option = None; + while let Some(next) = inbound.next().await { + match next { + Ok(frame) => { + let Some(openshell_core::proto::relay_frame::Payload::Data(data)) = frame.payload + else { + inbound_err = Some("relay inbound received non-data frame".to_string()); + break; + }; + if data.is_empty() { + continue; + } + if let Err(e) = ssh_w.write_all(&data).await { + inbound_err = Some(format!("write to ssh failed: {e}")); + break; + } + } + Err(e) => { + inbound_err = Some(format!("relay inbound errored: {e}")); + break; + } + } + } + + // Half-close the SSH socket's write side so the daemon sees EOF. + let _ = ssh_w.shutdown().await; + + // Dropping out_tx closes the outbound gRPC stream, letting the gateway + // observe EOF on its side too. + drop(out_tx); + let _ = ssh_to_grpc.await; + + if let Some(e) = inbound_err { + return Err(e.into()); + } + Ok(()) +} + +#[cfg(test)] +mod ocsf_event_tests { + use super::*; + + fn ctx() -> SandboxContext { + SandboxContext { + sandbox_id: "sbx-1".into(), + sandbox_name: "sandbox".into(), + container_image: "img".into(), + hostname: "host".into(), + product_version: "0.0.1".into(), + proxy_ip: "127.0.0.1".parse().unwrap(), + proxy_port: 3128, + } + } + + #[test] + fn gateway_endpoint_parses_https_with_port() { + let e = ocsf_gateway_endpoint("https://gateway.openshell:8443"); + assert_eq!(e.domain.as_deref(), Some("gateway.openshell")); + assert_eq!(e.port, Some(8443)); + } + + #[test] + fn gateway_endpoint_parses_http_with_port_and_path() { + let e = ocsf_gateway_endpoint("http://gw:7000/grpc"); + assert_eq!(e.domain.as_deref(), Some("gw")); + assert_eq!(e.port, Some(7000)); + } + + #[test] + fn gateway_endpoint_falls_back_without_port() { + let e = ocsf_gateway_endpoint("gateway.openshell"); + assert_eq!(e.domain.as_deref(), Some("gateway.openshell")); + assert_eq!(e.port, Some(0)); + } + + fn network_activity(event: &OcsfEvent) -> &openshell_ocsf::NetworkActivityEvent { + match event { + OcsfEvent::NetworkActivity(n) => n, + other => panic!("expected NetworkActivity, got {other:?}"), + } + } + + #[test] + fn session_established_emits_network_open_success() { + let event = session_established_event(&ctx(), "https://gw:443", "sess-1", 30); + let na = network_activity(&event); + assert_eq!(na.base.activity_id, ActivityId::Open.as_u8()); + assert_eq!(na.base.severity, SeverityId::Informational); + assert_eq!(na.base.status, Some(StatusId::Success)); + assert_eq!( + na.dst_endpoint.as_ref().and_then(|e| e.domain.as_deref()), + Some("gw") + ); + let msg = na.base.message.as_deref().unwrap_or_default(); + assert!(msg.contains("sess-1"), "message missing session_id: {msg}"); + assert!(msg.contains("heartbeat_secs=30"), "message: {msg}"); + } + + #[test] + fn session_closed_emits_network_close_success() { + let event = session_closed_event(&ctx(), "https://gw:443", "sbx-1"); + let na = network_activity(&event); + assert_eq!(na.base.activity_id, ActivityId::Close.as_u8()); + assert_eq!(na.base.severity, SeverityId::Informational); + assert_eq!(na.base.status, Some(StatusId::Success)); + } + + #[test] + fn session_failed_emits_network_fail_low() { + let event = session_failed_event(&ctx(), "https://gw:443", 3, "connect refused"); + let na = network_activity(&event); + assert_eq!(na.base.activity_id, ActivityId::Fail.as_u8()); + assert_eq!(na.base.severity, SeverityId::Low); + assert_eq!(na.base.status, Some(StatusId::Failure)); + let msg = na.base.message.as_deref().unwrap_or_default(); + assert!(msg.contains("attempt 3"), "message: {msg}"); + assert!(msg.contains("connect refused"), "message: {msg}"); + } + + #[test] + fn relay_open_emits_network_open_success() { + let event = relay_open_event(&ctx(), "ch-42"); + let na = network_activity(&event); + assert_eq!(na.base.activity_id, ActivityId::Open.as_u8()); + assert_eq!(na.base.severity, SeverityId::Informational); + assert!( + na.base + .message + .as_deref() + .unwrap_or_default() + .contains("ch-42") + ); + } + + #[test] + fn relay_closed_emits_network_close_success() { + let event = relay_closed_event(&ctx(), "ch-42"); + let na = network_activity(&event); + assert_eq!(na.base.activity_id, ActivityId::Close.as_u8()); + assert_eq!(na.base.status, Some(StatusId::Success)); + } + + #[test] + fn relay_failed_emits_network_fail_low() { + let event = relay_failed_event(&ctx(), "ch-42", "write to ssh failed"); + let na = network_activity(&event); + assert_eq!(na.base.activity_id, ActivityId::Fail.as_u8()); + assert_eq!(na.base.severity, SeverityId::Low); + assert_eq!(na.base.status, Some(StatusId::Failure)); + let msg = na.base.message.as_deref().unwrap_or_default(); + assert!(msg.contains("ch-42"), "message: {msg}"); + assert!(msg.contains("write to ssh failed"), "message: {msg}"); + } + + #[test] + fn relay_close_from_gateway_is_network_close_informational() { + let event = relay_close_from_gateway_event(&ctx(), "ch-42", "sandbox deleted"); + let na = network_activity(&event); + assert_eq!(na.base.activity_id, ActivityId::Close.as_u8()); + assert_eq!(na.base.severity, SeverityId::Informational); + let msg = na.base.message.as_deref().unwrap_or_default(); + assert!(msg.contains("sandbox deleted"), "message: {msg}"); + } + + #[test] + fn map_stream_message_treats_eof_as_reconnectable_error() { + let err = map_stream_message::(Ok(None), "gateway closed stream") + .expect_err("eof should force reconnect"); + assert_eq!(err.to_string(), "gateway closed stream"); + } +} diff --git a/crates/openshell-sandbox/tests/stdout_logging.rs b/crates/openshell-sandbox/tests/stdout_logging.rs new file mode 100644 index 0000000000..c4f5213de8 --- /dev/null +++ b/crates/openshell-sandbox/tests/stdout_logging.rs @@ -0,0 +1,41 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::process::Command; + +#[test] +fn startup_logs_go_to_stderr_not_stdout() { + let output = Command::new(env!("CARGO_BIN_EXE_openshell-sandbox")) + .arg("--") + .arg("/usr/bin/printf") + .arg("hello") + .env("OPENSHELL_LOG_LEVEL", "info") + .env_remove("RUST_LOG") + .env_remove("OPENSHELL_POLICY_RULES") + .env_remove("OPENSHELL_POLICY_DATA") + .env_remove("OPENSHELL_SANDBOX_ID") + .env_remove("OPENSHELL_ENDPOINT") + .output() + .expect("spawn openshell-sandbox"); + + assert!( + !output.status.success(), + "expected sandbox startup to fail without a policy source" + ); + + let stdout = String::from_utf8_lossy(&output.stdout); + let stderr = String::from_utf8_lossy(&output.stderr); + + assert!( + stdout.trim().is_empty(), + "expected startup logs on stderr only, got stdout: {stdout}" + ); + assert!( + stderr.contains("Starting sandbox"), + "expected startup log on stderr, got: {stderr}" + ); + assert!( + stderr.contains("Sandbox policy required"), + "expected missing-policy error on stderr, got: {stderr}" + ); +} diff --git a/crates/openshell-sandbox/tests/system_inference.rs b/crates/openshell-sandbox/tests/system_inference.rs index 5d581fbe23..20c39f3b66 100644 --- a/crates/openshell-sandbox/tests/system_inference.rs +++ b/crates/openshell-sandbox/tests/system_inference.rs @@ -20,6 +20,7 @@ fn make_system_route() -> ResolvedRoute { protocols: vec!["openai_chat_completions".to_string()], auth: AuthHeader::Bearer, default_headers: Vec::new(), + passthrough_headers: Vec::new(), timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, } } @@ -33,6 +34,7 @@ fn make_user_route() -> ResolvedRoute { protocols: vec!["openai_chat_completions".to_string()], auth: AuthHeader::Bearer, default_headers: Vec::new(), + passthrough_headers: Vec::new(), timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, } } @@ -126,6 +128,10 @@ async fn system_inference_with_anthropic_protocol() { protocols: vec!["anthropic_messages".to_string()], auth: AuthHeader::Custom("x-api-key"), default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], + passthrough_headers: vec![ + "anthropic-version".to_string(), + "anthropic-beta".to_string(), + ], timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, }; diff --git a/crates/openshell-sandbox/tests/websocket_upgrade.rs b/crates/openshell-sandbox/tests/websocket_upgrade.rs index ec226c9cf6..e4cd232ce1 100644 --- a/crates/openshell-sandbox/tests/websocket_upgrade.rs +++ b/crates/openshell-sandbox/tests/websocket_upgrade.rs @@ -1,6 +1,13 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +#![allow( + clippy::match_same_arms, + clippy::cast_possible_truncation, + clippy::doc_markdown, + reason = "Integration test: dual-arm patterns and small-payload casts are idiomatic in test setup." +)] + //! Integration test: WebSocket upgrade through the L7 relay. //! //! Spins up a dummy WebSocket echo server, connects a client through the @@ -111,7 +118,7 @@ async fn websocket_upgrade_through_l7_relay_exchanges_message() { // Run the relay in a background task (simulates what relay_rest does) let relay_handle = tokio::spawn(async move { - let outcome = RestProvider + let outcome = RestProvider::default() .relay(&req, &mut client_proxy, &mut upstream) .await .expect("relay should succeed"); @@ -239,7 +246,7 @@ async fn normal_http_request_still_works_after_relay_changes() { let outcome = tokio::time::timeout( std::time::Duration::from_secs(5), - RestProvider.relay(&req, &mut client_proxy, &mut upstream), + RestProvider::default().relay(&req, &mut client_proxy, &mut upstream), ) .await .expect("should not deadlock") diff --git a/crates/openshell-server/Cargo.toml b/crates/openshell-server/Cargo.toml index b4e8b9e2f5..cb6561f3e5 100644 --- a/crates/openshell-server/Cargo.toml +++ b/crates/openshell-server/Cargo.toml @@ -11,13 +11,17 @@ license.workspace = true repository.workspace = true [[bin]] -name = "openshell-server" +name = "openshell-gateway" path = "src/main.rs" [dependencies] openshell-core = { path = "../openshell-core" } +openshell-driver-docker = { path = "../openshell-driver-docker" } openshell-driver-kubernetes = { path = "../openshell-driver-kubernetes" } +openshell-driver-podman = { path = "../openshell-driver-podman" } +openshell-ocsf = { path = "../openshell-ocsf" } openshell-policy = { path = "../openshell-policy" } +openshell-providers = { path = "../openshell-providers" } openshell-router = { path = "../openshell-router" } # Async runtime @@ -55,6 +59,10 @@ anyhow = { workspace = true } tracing = { workspace = true } tracing-subscriber = { workspace = true } +# Metrics +metrics = { workspace = true } +metrics-exporter-prometheus = { workspace = true } + # Utilities futures = { workspace = true } bytes = { workspace = true } @@ -66,12 +74,14 @@ sqlx = { workspace = true } reqwest = { workspace = true } uuid = { workspace = true } hmac = "0.12" -sha2 = "0.10" +sha2 = { workspace = true } +jsonwebtoken = { workspace = true } hex = "0.4" russh = "0.57" -rand = "0.9" +rand = { workspace = true } petname = "2" ipnet = "2" +tempfile = "3" [features] dev-settings = ["openshell-core/dev-settings"] @@ -79,7 +89,6 @@ dev-settings = ["openshell-core/dev-settings"] [dev-dependencies] hyper-rustls = { version = "0.27", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "ring", "webpki-tokio"] } rcgen = { version = "0.13", features = ["crypto", "pem"] } -tempfile = "3" tokio-tungstenite = { workspace = true } futures-util = "0.3" wiremock = "0.6" diff --git a/crates/openshell-server/migrations/postgres/001_create_objects.sql b/crates/openshell-server/migrations/postgres/001_create_objects.sql index 0d2cb17da8..3978f84567 100644 --- a/crates/openshell-server/migrations/postgres/001_create_objects.sql +++ b/crates/openshell-server/migrations/postgres/001_create_objects.sql @@ -1,10 +1,29 @@ CREATE TABLE IF NOT EXISTS objects ( - object_type TEXT NOT NULL, - id TEXT NOT NULL, - name TEXT NOT NULL, - payload BYTEA NOT NULL, + id TEXT PRIMARY KEY, + object_type TEXT NOT NULL, + name TEXT, + scope TEXT, + version BIGINT, + status TEXT, + dedup_key TEXT, + hit_count BIGINT NOT NULL DEFAULT 0, + payload BYTEA NOT NULL, created_at_ms BIGINT NOT NULL, - updated_at_ms BIGINT NOT NULL, - PRIMARY KEY (id), - UNIQUE (object_type, name) + updated_at_ms BIGINT NOT NULL ); + +CREATE UNIQUE INDEX IF NOT EXISTS objects_name_uq + ON objects (object_type, name) + WHERE name IS NOT NULL; + +CREATE UNIQUE INDEX IF NOT EXISTS objects_version_uq + ON objects (object_type, scope, version) + WHERE scope IS NOT NULL AND version IS NOT NULL; + +CREATE INDEX IF NOT EXISTS objects_scope_status_idx + ON objects (object_type, scope, status, version) + WHERE scope IS NOT NULL; + +CREATE UNIQUE INDEX IF NOT EXISTS objects_dedup_uq + ON objects (object_type, scope, dedup_key) + WHERE dedup_key IS NOT NULL; diff --git a/crates/openshell-server/migrations/postgres/002_create_sandbox_policies.sql b/crates/openshell-server/migrations/postgres/002_create_sandbox_policies.sql deleted file mode 100644 index 1fdd49b05a..0000000000 --- a/crates/openshell-server/migrations/postgres/002_create_sandbox_policies.sql +++ /dev/null @@ -1,15 +0,0 @@ -CREATE TABLE IF NOT EXISTS sandbox_policies ( - id TEXT PRIMARY KEY, - sandbox_id TEXT NOT NULL, - version INTEGER NOT NULL, - policy_payload BYTEA NOT NULL, - policy_hash TEXT NOT NULL, - status TEXT NOT NULL DEFAULT 'pending', - load_error TEXT, - created_at_ms BIGINT NOT NULL, - loaded_at_ms BIGINT, - UNIQUE (sandbox_id, version) -); - -CREATE INDEX IF NOT EXISTS idx_sandbox_policies_lookup - ON sandbox_policies (sandbox_id, version DESC); diff --git a/crates/openshell-server/migrations/postgres/003_create_policy_recommendations.sql b/crates/openshell-server/migrations/postgres/003_create_policy_recommendations.sql deleted file mode 100644 index 0c6fa09eeb..0000000000 --- a/crates/openshell-server/migrations/postgres/003_create_policy_recommendations.sql +++ /dev/null @@ -1,33 +0,0 @@ --- Draft policy chunks: proposed network policy rules awaiting user approval. --- --- One row per (sandbox_id, host, port, binary). The toggle model allows: --- pending -> approved | rejected (initial decision) --- approved <-> rejected (toggle via approve/revoke) --- --- Upserts bump hit_count / last_seen_ms when the same denial recurs. -CREATE TABLE IF NOT EXISTS draft_policy_chunks ( - id TEXT PRIMARY KEY, - sandbox_id TEXT NOT NULL, - draft_version BIGINT NOT NULL, - status TEXT NOT NULL DEFAULT 'pending', - rule_name TEXT NOT NULL, - proposed_rule BYTEA NOT NULL, - rationale TEXT NOT NULL DEFAULT '', - security_notes TEXT NOT NULL DEFAULT '', - confidence DOUBLE PRECISION NOT NULL DEFAULT 0.0, - host TEXT NOT NULL DEFAULT '', - port INTEGER NOT NULL DEFAULT 0, - binary TEXT NOT NULL DEFAULT '', - hit_count INTEGER NOT NULL DEFAULT 1, - first_seen_ms BIGINT NOT NULL, - last_seen_ms BIGINT NOT NULL, - created_at_ms BIGINT NOT NULL, - decided_at_ms BIGINT -); - -CREATE INDEX IF NOT EXISTS idx_draft_chunks_sandbox - ON draft_policy_chunks (sandbox_id, status); - -CREATE UNIQUE INDEX IF NOT EXISTS idx_draft_chunks_endpoint - ON draft_policy_chunks (sandbox_id, host, port, binary) - WHERE status IN ('pending', 'approved', 'rejected'); diff --git a/crates/openshell-server/migrations/postgres/004_add_object_metadata.sql b/crates/openshell-server/migrations/postgres/004_add_object_metadata.sql new file mode 100644 index 0000000000..1f222469b5 --- /dev/null +++ b/crates/openshell-server/migrations/postgres/004_add_object_metadata.sql @@ -0,0 +1,5 @@ +-- Add labels column to support Kubernetes-style object metadata +ALTER TABLE objects ADD COLUMN labels JSONB; + +-- Backfill existing rows with empty labels +UPDATE objects SET labels = '{}'::jsonb WHERE labels IS NULL; diff --git a/crates/openshell-server/migrations/sqlite/001_create_objects.sql b/crates/openshell-server/migrations/sqlite/001_create_objects.sql index 26a77e9da6..749a6460e6 100644 --- a/crates/openshell-server/migrations/sqlite/001_create_objects.sql +++ b/crates/openshell-server/migrations/sqlite/001_create_objects.sql @@ -1,10 +1,29 @@ CREATE TABLE IF NOT EXISTS objects ( - object_type TEXT NOT NULL, - id TEXT NOT NULL, - name TEXT NOT NULL, - payload BLOB NOT NULL, + id TEXT PRIMARY KEY, + object_type TEXT NOT NULL, + name TEXT, + scope TEXT, + version INTEGER, + status TEXT, + dedup_key TEXT, + hit_count INTEGER NOT NULL DEFAULT 0, + payload BLOB NOT NULL, created_at_ms INTEGER NOT NULL, - updated_at_ms INTEGER NOT NULL, - PRIMARY KEY (id), - UNIQUE (object_type, name) + updated_at_ms INTEGER NOT NULL ); + +CREATE UNIQUE INDEX IF NOT EXISTS objects_name_uq + ON objects (object_type, name) + WHERE name IS NOT NULL; + +CREATE UNIQUE INDEX IF NOT EXISTS objects_version_uq + ON objects (object_type, scope, version) + WHERE scope IS NOT NULL AND version IS NOT NULL; + +CREATE INDEX IF NOT EXISTS objects_scope_status_idx + ON objects (object_type, scope, status, version) + WHERE scope IS NOT NULL; + +CREATE UNIQUE INDEX IF NOT EXISTS objects_dedup_uq + ON objects (object_type, scope, dedup_key) + WHERE dedup_key IS NOT NULL; diff --git a/crates/openshell-server/migrations/sqlite/002_create_sandbox_policies.sql b/crates/openshell-server/migrations/sqlite/002_create_sandbox_policies.sql deleted file mode 100644 index 395b1897a9..0000000000 --- a/crates/openshell-server/migrations/sqlite/002_create_sandbox_policies.sql +++ /dev/null @@ -1,15 +0,0 @@ -CREATE TABLE IF NOT EXISTS sandbox_policies ( - id TEXT PRIMARY KEY, - sandbox_id TEXT NOT NULL, - version INTEGER NOT NULL, - policy_payload BLOB NOT NULL, - policy_hash TEXT NOT NULL, - status TEXT NOT NULL DEFAULT 'pending', - load_error TEXT, - created_at_ms INTEGER NOT NULL, - loaded_at_ms INTEGER, - UNIQUE (sandbox_id, version) -); - -CREATE INDEX IF NOT EXISTS idx_sandbox_policies_lookup - ON sandbox_policies (sandbox_id, version DESC); diff --git a/crates/openshell-server/migrations/sqlite/003_create_policy_recommendations.sql b/crates/openshell-server/migrations/sqlite/003_create_policy_recommendations.sql deleted file mode 100644 index 069ce20480..0000000000 --- a/crates/openshell-server/migrations/sqlite/003_create_policy_recommendations.sql +++ /dev/null @@ -1,35 +0,0 @@ --- Draft policy chunks: proposed network policy rules awaiting user approval. --- --- One row per (sandbox_id, host, port, binary). The toggle model allows: --- pending -> approved | rejected (initial decision) --- approved <-> rejected (toggle via approve/revoke) --- --- Upserts bump hit_count / last_seen_ms when the same denial recurs. -CREATE TABLE IF NOT EXISTS draft_policy_chunks ( - id TEXT PRIMARY KEY, - sandbox_id TEXT NOT NULL, - draft_version INTEGER NOT NULL, - status TEXT NOT NULL DEFAULT 'pending', - rule_name TEXT NOT NULL, - proposed_rule BLOB NOT NULL, - rationale TEXT NOT NULL DEFAULT '', - security_notes TEXT NOT NULL DEFAULT '', - confidence REAL NOT NULL DEFAULT 0.0, - host TEXT NOT NULL DEFAULT '', - port INTEGER NOT NULL DEFAULT 0, - binary TEXT NOT NULL DEFAULT '', - hit_count INTEGER NOT NULL DEFAULT 1, - first_seen_ms INTEGER NOT NULL, - last_seen_ms INTEGER NOT NULL, - created_at_ms INTEGER NOT NULL, - decided_at_ms INTEGER -); - -CREATE INDEX IF NOT EXISTS idx_draft_chunks_sandbox - ON draft_policy_chunks (sandbox_id, status); - --- Only one active chunk per (sandbox, endpoint, binary). Covers all three --- statuses so rejected chunks block duplicate proposals until re-approved. -CREATE UNIQUE INDEX IF NOT EXISTS idx_draft_chunks_endpoint - ON draft_policy_chunks (sandbox_id, host, port, binary) - WHERE status IN ('pending', 'approved', 'rejected'); diff --git a/crates/openshell-server/migrations/sqlite/004_add_object_metadata.sql b/crates/openshell-server/migrations/sqlite/004_add_object_metadata.sql new file mode 100644 index 0000000000..31536c1537 --- /dev/null +++ b/crates/openshell-server/migrations/sqlite/004_add_object_metadata.sql @@ -0,0 +1,5 @@ +-- Add labels column to support Kubernetes-style object metadata +ALTER TABLE objects ADD COLUMN labels TEXT; + +-- Backfill existing rows with empty labels +UPDATE objects SET labels = '{}' WHERE labels IS NULL; diff --git a/crates/openshell-server/src/auth/authz.rs b/crates/openshell-server/src/auth/authz.rs new file mode 100644 index 0000000000..05ac19354d --- /dev/null +++ b/crates/openshell-server/src/auth/authz.rs @@ -0,0 +1,521 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Authorization policy evaluation. +//! +//! Determines whether an authenticated identity is allowed to call a given +//! gRPC method. This module owns the RBAC policy — which methods require +//! which roles — while authentication providers (OIDC, mTLS, etc.) own +//! identity verification. +//! +//! This separation follows RFC 0001's control-plane identity design: +//! authentication is a driver concern, authorization is a gateway concern. + +use super::identity::Identity; +use tonic::Status; +use tracing::debug; + +/// gRPC methods that require the admin role. +/// All other authenticated methods require the user role. +const ADMIN_METHODS: &[&str] = &[ + // Provider management + "/openshell.v1.OpenShell/CreateProvider", + "/openshell.v1.OpenShell/UpdateProvider", + "/openshell.v1.OpenShell/DeleteProvider", + // Global config and policy + "/openshell.v1.OpenShell/UpdateConfig", + // Draft policy approvals + "/openshell.v1.OpenShell/ApproveDraftChunk", + "/openshell.v1.OpenShell/ApproveAllDraftChunks", + "/openshell.v1.OpenShell/RejectDraftChunk", + "/openshell.v1.OpenShell/EditDraftChunk", + "/openshell.v1.OpenShell/UndoDraftChunk", + "/openshell.v1.OpenShell/ClearDraftChunks", + // Cluster inference write + "/openshell.inference.v1.Inference/SetClusterInference", +]; + +/// Exhaustive mapping of Bearer-authenticated gRPC methods to required scopes. +/// Methods not listed here require `openshell:all` when scope enforcement is enabled. +const SCOPED_METHODS: &[(&str, &str)] = &[ + // sandbox:read + ("/openshell.v1.OpenShell/GetSandbox", "sandbox:read"), + ("/openshell.v1.OpenShell/ListSandboxes", "sandbox:read"), + ("/openshell.v1.OpenShell/WatchSandbox", "sandbox:read"), + ("/openshell.v1.OpenShell/GetSandboxLogs", "sandbox:read"), + ( + "/openshell.v1.OpenShell/GetSandboxPolicyStatus", + "sandbox:read", + ), + ( + "/openshell.v1.OpenShell/ListSandboxPolicies", + "sandbox:read", + ), + // sandbox:write + ("/openshell.v1.OpenShell/CreateSandbox", "sandbox:write"), + ("/openshell.v1.OpenShell/DeleteSandbox", "sandbox:write"), + ("/openshell.v1.OpenShell/ExecSandbox", "sandbox:write"), + ("/openshell.v1.OpenShell/CreateSshSession", "sandbox:write"), + ("/openshell.v1.OpenShell/RevokeSshSession", "sandbox:write"), + // provider:read + ("/openshell.v1.OpenShell/GetProvider", "provider:read"), + ("/openshell.v1.OpenShell/ListProviders", "provider:read"), + // provider:write + ("/openshell.v1.OpenShell/CreateProvider", "provider:write"), + ("/openshell.v1.OpenShell/UpdateProvider", "provider:write"), + ("/openshell.v1.OpenShell/DeleteProvider", "provider:write"), + // config:read + ("/openshell.v1.OpenShell/GetGatewayConfig", "config:read"), + ("/openshell.v1.OpenShell/GetSandboxConfig", "config:read"), + ("/openshell.v1.OpenShell/GetDraftPolicy", "config:read"), + ("/openshell.v1.OpenShell/GetDraftHistory", "config:read"), + // config:write + ("/openshell.v1.OpenShell/UpdateConfig", "config:write"), + ("/openshell.v1.OpenShell/ApproveDraftChunk", "config:write"), + ( + "/openshell.v1.OpenShell/ApproveAllDraftChunks", + "config:write", + ), + ("/openshell.v1.OpenShell/RejectDraftChunk", "config:write"), + ("/openshell.v1.OpenShell/EditDraftChunk", "config:write"), + ("/openshell.v1.OpenShell/UndoDraftChunk", "config:write"), + ("/openshell.v1.OpenShell/ClearDraftChunks", "config:write"), + // inference:read + ( + "/openshell.inference.v1.Inference/GetClusterInference", + "inference:read", + ), + // inference:write + ( + "/openshell.inference.v1.Inference/SetClusterInference", + "inference:write", + ), +]; + +const SCOPE_ALL: &str = "openshell:all"; + +/// Authorization policy configuration. +/// +/// Supports two modes: +/// - **RBAC mode**: both `admin_role` and `user_role` are non-empty. +/// - **Authentication-only mode**: both are empty (any valid token is authorized). +/// +/// Partial configuration (one empty, one set) is rejected at construction +/// to prevent accidentally leaving admin endpoints unprotected. +#[derive(Debug, Clone)] +pub struct AuthzPolicy { + /// Role name that grants admin access. Empty disables admin checks. + pub admin_role: String, + /// Role name that grants standard user access. Empty disables user checks. + pub user_role: String, + /// When true, enforce scope-based permissions on top of roles. + pub scopes_enabled: bool, +} + +impl AuthzPolicy { + /// Validate the policy configuration. + /// + /// Returns an error if only one of admin/user role is set — either + /// both must be set (RBAC mode) or both empty (auth-only mode). + pub fn validate(&self) -> Result<(), String> { + let admin_set = !self.admin_role.is_empty(); + let user_set = !self.user_role.is_empty(); + if admin_set != user_set { + return Err(format!( + "OIDC RBAC misconfiguration: admin_role={:?}, user_role={:?}. \ + Either set both roles (RBAC mode) or leave both empty (authentication-only mode).", + self.admin_role, self.user_role, + )); + } + Ok(()) + } +} + +impl AuthzPolicy { + /// Check whether the identity is authorized to call the given method. + /// + /// Returns `Ok(())` if authorized, `Err(PERMISSION_DENIED)` if not. + /// When both role names are empty, all authenticated callers are authorized + /// (authentication-only mode for providers like GitHub). + #[allow(clippy::result_large_err)] + pub fn check(&self, identity: &Identity, method: &str) -> Result<(), Status> { + let required = if ADMIN_METHODS.contains(&method) { + &self.admin_role + } else { + &self.user_role + }; + + // Empty role name = skip role check for this level (auth-only mode). + // Scope enforcement still applies if enabled. + if !required.is_empty() { + // Admin role implicitly satisfies user role requirements. + let has_role = identity.roles.iter().any(|r| r == required) + || (!self.admin_role.is_empty() + && required == &self.user_role + && identity.roles.iter().any(|r| r == &self.admin_role)); + + if !has_role { + debug!( + sub = %identity.subject, + required_role = required, + user_roles = ?identity.roles, + method = method, + "authorization denied: missing role" + ); + return Err(Status::permission_denied(format!( + "role '{required}' required" + ))); + } + } + + if self.scopes_enabled { + self.check_scope(identity, method)?; + } + + Ok(()) + } + + #[allow(clippy::result_large_err, clippy::unused_self)] + fn check_scope(&self, identity: &Identity, method: &str) -> Result<(), Status> { + if identity.scopes.iter().any(|s| s == SCOPE_ALL) { + return Ok(()); + } + + let required_scope = SCOPED_METHODS + .iter() + .find(|(m, _)| *m == method) + .map_or(SCOPE_ALL, |(_, s)| *s); + + if identity.scopes.iter().any(|s| s == required_scope) { + return Ok(()); + } + + debug!( + sub = %identity.subject, + required_scope = required_scope, + user_scopes = ?identity.scopes, + method = method, + "authorization denied: missing scope" + ); + Err(Status::permission_denied(format!( + "scope '{required_scope}' required" + ))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::auth::identity::IdentityProvider; + + fn default_policy() -> AuthzPolicy { + AuthzPolicy { + admin_role: "openshell-admin".to_string(), + user_role: "openshell-user".to_string(), + scopes_enabled: false, + } + } + + fn scoped_policy() -> AuthzPolicy { + AuthzPolicy { + admin_role: "openshell-admin".to_string(), + user_role: "openshell-user".to_string(), + scopes_enabled: true, + } + } + + fn identity_with_roles(roles: &[&str]) -> Identity { + Identity { + subject: "test-user".to_string(), + display_name: None, + roles: roles.iter().map(|r| (*r).to_string()).collect(), + scopes: vec![], + provider: IdentityProvider::Oidc, + } + } + + fn identity_with_roles_and_scopes(roles: &[&str], scopes: &[&str]) -> Identity { + Identity { + subject: "test-user".to_string(), + display_name: None, + roles: roles.iter().map(|r| (*r).to_string()).collect(), + scopes: scopes.iter().map(|s| (*s).to_string()).collect(), + provider: IdentityProvider::Oidc, + } + } + + #[test] + fn user_can_access_user_methods() { + let id = identity_with_roles(&["openshell-user"]); + let policy = default_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + } + + #[test] + fn user_cannot_access_admin_methods() { + let id = identity_with_roles(&["openshell-user"]); + let policy = default_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/CreateProvider") + .is_err() + ); + } + + #[test] + fn admin_can_access_admin_methods() { + let id = identity_with_roles(&["openshell-admin", "openshell-user"]); + let policy = default_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/CreateProvider") + .is_ok() + ); + } + + #[test] + fn admin_only_can_access_user_methods() { + let id = identity_with_roles(&["openshell-admin"]); + let policy = default_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + } + + #[test] + fn empty_roles_rejected() { + let id = identity_with_roles(&[]); + let policy = default_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_err() + ); + } + + #[test] + fn empty_role_names_skip_rbac() { + let id = identity_with_roles(&[]); + let policy = AuthzPolicy { + admin_role: String::new(), + user_role: String::new(), + scopes_enabled: false, + }; + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/CreateProvider") + .is_ok() + ); + } + + #[test] + fn custom_role_names() { + let id = identity_with_roles(&["OpenShell.Admin", "OpenShell.User"]); + let policy = AuthzPolicy { + admin_role: "OpenShell.Admin".to_string(), + user_role: "OpenShell.User".to_string(), + scopes_enabled: false, + }; + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/CreateProvider") + .is_ok() + ); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + } + + #[test] + fn validate_accepts_both_roles_set() { + let policy = default_policy(); + assert!(policy.validate().is_ok()); + } + + #[test] + fn validate_accepts_both_roles_empty() { + let policy = AuthzPolicy { + admin_role: String::new(), + user_role: String::new(), + scopes_enabled: false, + }; + assert!(policy.validate().is_ok()); + } + + #[test] + fn validate_rejects_partial_empty_admin_only() { + let policy = AuthzPolicy { + admin_role: "admin".to_string(), + user_role: String::new(), + scopes_enabled: false, + }; + assert!(policy.validate().is_err()); + } + + #[test] + fn validate_rejects_partial_empty_user_only() { + let policy = AuthzPolicy { + admin_role: String::new(), + user_role: "user".to_string(), + scopes_enabled: false, + }; + assert!(policy.validate().is_err()); + } + + // ---- Scope enforcement tests ---- + + #[test] + fn scopes_disabled_skips_scope_check() { + let id = identity_with_roles(&["openshell-user"]); + let policy = default_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + } + + #[test] + fn scoped_access_allowed() { + let id = + identity_with_roles_and_scopes(&["openshell-user"], &["sandbox:read", "sandbox:write"]); + let policy = scoped_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/CreateSandbox") + .is_ok() + ); + } + + #[test] + fn scoped_access_denied() { + let id = identity_with_roles_and_scopes(&["openshell-user"], &["sandbox:read"]); + let policy = scoped_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + let err = policy + .check(&id, "/openshell.v1.OpenShell/CreateSandbox") + .unwrap_err(); + assert_eq!(err.code(), tonic::Code::PermissionDenied); + assert!(err.message().contains("sandbox:write")); + } + + #[test] + fn get_sandbox_config_requires_config_read_scope() { + let policy = scoped_policy(); + let id = identity_with_roles_and_scopes(&["openshell-user"], &["config:read"]); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/GetSandboxConfig") + .is_ok() + ); + + let wrong_scope = identity_with_roles_and_scopes(&["openshell-user"], &["sandbox:read"]); + let err = policy + .check(&wrong_scope, "/openshell.v1.OpenShell/GetSandboxConfig") + .unwrap_err(); + assert_eq!(err.code(), tonic::Code::PermissionDenied); + assert!(err.message().contains("config:read")); + } + + #[test] + fn no_openshell_scopes_denied() { + let id = identity_with_roles_and_scopes(&["openshell-user"], &[]); + let policy = scoped_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_err() + ); + } + + #[test] + fn openshell_all_with_user_role() { + let id = identity_with_roles_and_scopes(&["openshell-user"], &["openshell:all"]); + let policy = scoped_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/GetProvider") + .is_ok() + ); + // admin methods still denied by role check + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/CreateProvider") + .is_err() + ); + } + + #[test] + fn openshell_all_with_admin_role() { + let id = identity_with_roles_and_scopes(&["openshell-admin"], &["openshell:all"]); + let policy = scoped_policy(); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/CreateProvider") + .is_ok() + ); + assert!( + policy + .check(&id, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + } + + #[test] + fn unknown_method_requires_openshell_all() { + let id = identity_with_roles_and_scopes(&["openshell-user"], &["sandbox:read"]); + let policy = scoped_policy(); + let err = policy + .check(&id, "/openshell.v1.OpenShell/SomeFutureMethod") + .unwrap_err(); + assert!(err.message().contains("openshell:all")); + } + + #[test] + fn auth_only_mode_with_scopes_still_enforces_scopes() { + let policy = AuthzPolicy { + admin_role: String::new(), + user_role: String::new(), + scopes_enabled: true, + }; + let id_with_scope = identity_with_roles_and_scopes(&[], &["sandbox:read"]); + assert!( + policy + .check(&id_with_scope, "/openshell.v1.OpenShell/ListSandboxes") + .is_ok() + ); + let id_without_scope = identity_with_roles_and_scopes(&[], &[]); + assert!( + policy + .check(&id_without_scope, "/openshell.v1.OpenShell/ListSandboxes") + .is_err() + ); + } +} diff --git a/crates/openshell-server/src/auth.rs b/crates/openshell-server/src/auth/http.rs similarity index 92% rename from crates/openshell-server/src/auth.rs rename to crates/openshell-server/src/auth/http.rs index b896d062c6..f0e7011658 100644 --- a/crates/openshell-server/src/auth.rs +++ b/crates/openshell-server/src/auth/http.rs @@ -16,9 +16,9 @@ //! CLI's ephemeral localhost server captures and stores the token. use axum::{ - Router, + Json, Router, extract::{Query, State}, - http::HeaderMap, + http::{HeaderMap, StatusCode}, response::{Html, IntoResponse}, routing::get, }; @@ -58,9 +58,27 @@ struct ConnectParams { pub fn router(state: Arc) -> Router { Router::new() .route("/auth/connect", get(auth_connect)) + .route("/auth/oidc-config", get(oidc_config_handler)) .with_state(state) } +/// OIDC configuration discovery endpoint. +/// +/// Returns the OIDC issuer and audience when OIDC is configured on the server, +/// so CLI clients can auto-discover settings during `gateway add`. +async fn oidc_config_handler(State(state): State>) -> impl IntoResponse { + state.config.oidc.as_ref().map_or_else( + || StatusCode::NOT_FOUND.into_response(), + |oidc| { + Json(serde_json::json!({ + "issuer": oidc.issuer, + "audience": oidc.audience, + })) + .into_response() + }, + ) +} + /// Handle the auth connect request. /// /// Reads `CF_Authorization` from the cookie header (server-side extraction @@ -96,32 +114,29 @@ async fn auth_connect( let safe_gateway = html_escape(&gateway_display); - match cf_token { - Some(token) => { - let nonce = uuid::Uuid::new_v4().to_string(); - let csp = format!( - "default-src 'none'; script-src 'nonce-{nonce}'; style-src 'unsafe-inline'; connect-src http://127.0.0.1:*" - ); - ( - [(header::CONTENT_SECURITY_POLICY, csp)], - Html(render_connect_page( - &safe_gateway, - params.callback_port, - &token, - ¶ms.code, - &nonce, - )), - ) - .into_response() - } - None => { - let csp = "default-src 'none'; style-src 'unsafe-inline'".to_string(); - ( - [(header::CONTENT_SECURITY_POLICY, csp)], - Html(render_waiting_page(params.callback_port, ¶ms.code)), - ) - .into_response() - } + if let Some(token) = cf_token { + let nonce = uuid::Uuid::new_v4().to_string(); + let csp = format!( + "default-src 'none'; script-src 'nonce-{nonce}'; style-src 'unsafe-inline'; connect-src http://127.0.0.1:*" + ); + ( + [(header::CONTENT_SECURITY_POLICY, csp)], + Html(render_connect_page( + &safe_gateway, + params.callback_port, + &token, + ¶ms.code, + &nonce, + )), + ) + .into_response() + } else { + let csp = "default-src 'none'; style-src 'unsafe-inline'".to_string(); + ( + [(header::CONTENT_SECURITY_POLICY, csp)], + Html(render_waiting_page(params.callback_port, ¶ms.code)), + ) + .into_response() } } diff --git a/crates/openshell-server/src/auth/identity.rs b/crates/openshell-server/src/auth/identity.rs new file mode 100644 index 0000000000..8c3da3cc3f --- /dev/null +++ b/crates/openshell-server/src/auth/identity.rs @@ -0,0 +1,44 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Provider-agnostic identity representation. +//! +//! Any authentication backend (OIDC, mTLS, static RBAC, OS users) produces +//! an `Identity` that the authorization layer can evaluate without knowing +//! which provider authenticated the caller. + +/// Authenticated caller identity. +/// +/// Produced by an authentication provider and consumed by the authorization +/// layer. The gateway's auth middleware converts provider-specific claims +/// (OIDC JWT, mTLS cert CN, etc.) into this common representation. +#[derive(Debug, Clone)] +pub struct Identity { + /// Unique subject identifier (OIDC `sub`, cert CN, username, etc.). + pub subject: String, + + /// Human-readable display name (OIDC `preferred_username`, cert CN, etc.). + pub display_name: Option, + + /// Roles granted to this identity (OIDC `realm_access.roles`, cert OU, etc.). + pub roles: Vec, + + /// `OAuth2` scopes granted to this identity. Empty when scope enforcement is disabled. + pub scopes: Vec, + + /// Which authentication provider produced this identity. + pub provider: IdentityProvider, +} + +/// Authentication provider that produced an identity. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum IdentityProvider { + /// OIDC/OAuth2 JWT bearer token. + Oidc, + /// mTLS client certificate. + Mtls, + /// Cloudflare Access JWT. + CloudflareAccess, + /// Internal (skip-listed methods, sandbox supervisor RPCs). + Internal, +} diff --git a/crates/openshell-server/src/auth/mod.rs b/crates/openshell-server/src/auth/mod.rs new file mode 100644 index 0000000000..8e4f332d8d --- /dev/null +++ b/crates/openshell-server/src/auth/mod.rs @@ -0,0 +1,16 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Authentication and authorization for the gateway server. +//! +//! - `oidc`: JWT validation against OIDC providers (Keycloak, Entra ID, Okta) +//! - `authz`: Role-based and scope-based access control +//! - `identity`: Provider-agnostic identity representation +//! - `http`: HTTP endpoints for auth discovery and token exchange + +pub mod authz; +mod http; +pub mod identity; +pub mod oidc; + +pub use http::router; diff --git a/crates/openshell-server/src/auth/oidc.rs b/crates/openshell-server/src/auth/oidc.rs new file mode 100644 index 0000000000..d3b74aa815 --- /dev/null +++ b/crates/openshell-server/src/auth/oidc.rs @@ -0,0 +1,606 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! OIDC JWT authentication provider. +//! +//! Validates `authorization: Bearer ` headers against a Keycloak (or +//! any OIDC-compliant) issuer using cached JWKS keys. Produces an +//! `Identity` that the authorization layer (`authz.rs`) evaluates. +//! +//! This module owns authentication (verifying who the caller is). +//! Authorization (deciding what the caller can do) is in `authz.rs`. + +use super::identity::{Identity, IdentityProvider}; +use jsonwebtoken::{Algorithm, DecodingKey, Validation, decode, decode_header}; +use openshell_core::OidcConfig; +use reqwest::Client; +use serde::Deserialize; +use std::collections::HashMap; +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tokio::sync::RwLock; +use tonic::Status; +use tracing::{debug, info, warn}; + +/// Internal metadata header set by the auth middleware after it validates +/// a sandbox-secret-authenticated request. This is stripped from all incoming +/// requests first so external callers cannot spoof it. +pub const INTERNAL_AUTH_SOURCE_HEADER: &str = "x-openshell-auth-source"; +/// Internal auth-source marker for requests authenticated via the shared +/// sandbox secret. +pub const AUTH_SOURCE_SANDBOX_SECRET: &str = "sandbox-secret"; + +/// Truly unauthenticated methods — health probes and infrastructure. +const UNAUTHENTICATED_METHODS: &[&str] = &[ + "/openshell.v1.OpenShell/Health", + "/openshell.inference.v1.Inference/Health", +]; + +/// Path prefixes that bypass OIDC validation (gRPC reflection, health probes). +const UNAUTHENTICATED_PREFIXES: &[&str] = &["/grpc.reflection.", "/grpc.health."]; + +/// Sandbox-to-server RPCs that use the shared sandbox secret instead of +/// OIDC Bearer tokens. These require the `x-sandbox-secret` metadata header +/// matching the server's SSH handshake secret. +const SANDBOX_SECRET_METHODS: &[&str] = &[ + "/openshell.v1.OpenShell/ReportPolicyStatus", + "/openshell.v1.OpenShell/PushSandboxLogs", + "/openshell.v1.OpenShell/GetSandboxProviderEnvironment", + "/openshell.v1.OpenShell/SubmitPolicyAnalysis", + "/openshell.sandbox.v1.SandboxService/GetSandboxConfig", + "/openshell.inference.v1.Inference/GetInferenceBundle", +]; + +/// Methods that accept either OIDC Bearer token (CLI users) or sandbox +/// secret (supervisor). `UpdateConfig` is called by both CLI +/// (policy/settings mutations) and the sandbox supervisor (policy sync on +/// startup). `OpenShell/GetSandboxConfig` serves CLI settings reads while +/// remaining compatible with sandbox-secret-authenticated callers. +const DUAL_AUTH_METHODS: &[&str] = &[ + "/openshell.v1.OpenShell/UpdateConfig", + "/openshell.v1.OpenShell/GetSandboxConfig", +]; + +/// Returns `true` if the method accepts either Bearer or sandbox-secret auth. +pub fn is_dual_auth_method(path: &str) -> bool { + DUAL_AUTH_METHODS.contains(&path) +} + +/// Returns `true` if the method needs no authentication at all. +pub fn is_unauthenticated_method(path: &str) -> bool { + UNAUTHENTICATED_METHODS.contains(&path) + || UNAUTHENTICATED_PREFIXES + .iter() + .any(|prefix| path.starts_with(prefix)) +} + +/// Returns `true` if the method authenticates via the sandbox shared secret +/// rather than an OIDC Bearer token. +pub fn is_sandbox_secret_method(path: &str) -> bool { + SANDBOX_SECRET_METHODS.contains(&path) +} + +/// Validate the `x-sandbox-secret` header against the server's handshake secret. +#[allow(clippy::result_large_err)] +pub fn validate_sandbox_secret( + headers: &http::HeaderMap, + expected_secret: &str, +) -> Result<(), Status> { + let provided = headers + .get("x-sandbox-secret") + .and_then(|v| v.to_str().ok()) + .ok_or_else(|| Status::unauthenticated("sandbox secret required for this method"))?; + + if provided != expected_secret { + return Err(Status::unauthenticated("invalid sandbox secret")); + } + + Ok(()) +} + +/// Remove internal auth-source markers from the request before any auth +/// decision is made so external callers cannot spoof them. +pub fn clear_internal_auth_markers(headers: &mut http::HeaderMap) { + headers.remove(INTERNAL_AUTH_SOURCE_HEADER); +} + +/// Mark the request as authenticated via the shared sandbox secret. +pub fn mark_sandbox_secret_authenticated(headers: &mut http::HeaderMap) { + headers.insert( + INTERNAL_AUTH_SOURCE_HEADER, + http::HeaderValue::from_static(AUTH_SOURCE_SANDBOX_SECRET), + ); +} + +/// Returns `true` if the request metadata indicates sandbox-secret auth. +pub fn is_sandbox_secret_authenticated(metadata: &tonic::metadata::MetadataMap) -> bool { + metadata + .get(INTERNAL_AUTH_SOURCE_HEADER) + .and_then(|v| v.to_str().ok()) + == Some(AUTH_SOURCE_SANDBOX_SECRET) +} + +/// Cached JWKS key set fetched from the OIDC issuer. +/// +/// A `refresh_mutex` ensures that only one refresh runs at a time, +/// preventing a "thundering herd" when the TTL expires or a new `kid` +/// is encountered under concurrent load. +pub struct JwksCache { + keys: Arc>>, + jwks_uri: String, + ttl: Duration, + last_refresh: Arc>, + /// Serializes JWKS refresh operations so concurrent requests coalesce + /// into a single HTTP fetch rather than stampeding the OIDC provider. + refresh_mutex: tokio::sync::Mutex<()>, + http: Client, + config: OidcConfig, +} + +impl std::fmt::Debug for JwksCache { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("JwksCache") + .field("jwks_uri", &self.jwks_uri) + .field("ttl", &self.ttl) + .finish() + } +} + +/// OIDC discovery document (subset of fields we need). +#[derive(Deserialize)] +struct OidcDiscovery { + issuer: String, + jwks_uri: String, +} + +/// JWKS key set. +#[derive(Deserialize)] +struct JwkSet { + keys: Vec, +} + +/// A single JWK key. +#[derive(Deserialize)] +struct JwkKey { + kid: Option, + kty: String, + #[serde(default)] + n: String, + #[serde(default)] + e: String, +} + +/// Claims extracted from a validated JWT. +#[derive(Debug, Deserialize)] +pub struct OidcClaims { + pub sub: String, + #[serde(default)] + pub preferred_username: Option, + #[serde(default)] + #[allow(dead_code)] + pub email: Option, + /// Roles extracted from the configurable claim path. + #[serde(skip)] + pub roles: Vec, + /// Raw claims for flexible role extraction. + #[serde(flatten)] + extra: serde_json::Value, +} + +const STANDARD_OIDC_SCOPES: &[&str] = &["openid", "profile", "email", "offline_access"]; + +impl OidcClaims { + /// Extract roles from the JWT claims using a dot-separated path. + /// + /// Supports paths like: + /// - `realm_access.roles` (Keycloak) + /// - `roles` (Entra ID) + /// - `groups` (Okta) + fn extract_roles(&mut self, roles_claim: &str) { + let mut value = &self.extra; + for segment in roles_claim.split('.') { + match value.get(segment) { + Some(v) => value = v, + None => return, + } + } + if let Some(arr) = value.as_array() { + self.roles = arr + .iter() + .filter_map(|v| v.as_str().map(String::from)) + .collect(); + } + } + + /// Extract scopes from the JWT claims using a dot-separated path. + /// + /// Handles two formats: + /// - Space-delimited string: `"openid sandbox:read sandbox:write"` (Keycloak, Entra) + /// - JSON array: `["sandbox:read", "sandbox:write"]` (Okta) + /// + /// Filters out standard OIDC scopes (`openid`, `profile`, `email`, `offline_access`). + fn extract_scopes(&self, scopes_claim: &str) -> Vec { + let mut value = &self.extra; + for segment in scopes_claim.split('.') { + match value.get(segment) { + Some(v) => value = v, + None => return vec![], + } + } + + let raw: Vec = if let Some(s) = value.as_str() { + s.split_whitespace().map(String::from).collect() + } else if let Some(arr) = value.as_array() { + arr.iter() + .filter_map(|v| v.as_str().map(String::from)) + .collect() + } else { + return vec![]; + }; + + raw.into_iter() + .filter(|s| !STANDARD_OIDC_SCOPES.contains(&s.as_str())) + .collect() + } +} + +impl JwksCache { + /// Create a new JWKS cache, discovering the JWKS URI and fetching the + /// initial key set. + pub async fn new(config: &OidcConfig) -> Result { + let http = Client::builder() + .timeout(Duration::from_secs(10)) + .build() + .map_err(|e| format!("failed to create HTTP client: {e}"))?; + + // Discover JWKS URI from the OIDC discovery endpoint. + let discovery_url = format!( + "{}/.well-known/openid-configuration", + config.issuer.trim_end_matches('/') + ); + info!(url = %discovery_url, "Discovering OIDC configuration"); + + let discovery: OidcDiscovery = http + .get(&discovery_url) + .send() + .await + .map_err(|e| format!("OIDC discovery request failed: {e}"))? + .json() + .await + .map_err(|e| format!("OIDC discovery response parse failed: {e}"))?; + + // Validate the discovery document's issuer matches our configured issuer. + let expected = config.issuer.trim_end_matches('/'); + let actual = discovery.issuer.trim_end_matches('/'); + if expected != actual { + return Err(format!( + "OIDC discovery issuer mismatch: expected '{expected}', got '{actual}'" + )); + } + + info!(jwks_uri = %discovery.jwks_uri, "OIDC JWKS URI discovered"); + + let cache = Self { + keys: Arc::new(RwLock::new(HashMap::new())), + jwks_uri: discovery.jwks_uri, + ttl: Duration::from_secs(config.jwks_ttl_secs), + last_refresh: Arc::new(RwLock::new( + Instant::now() + .checked_sub(Duration::from_secs(config.jwks_ttl_secs + 1)) + .unwrap_or_else(Instant::now), + )), + refresh_mutex: tokio::sync::Mutex::new(()), + http, + config: config.clone(), + }; + + cache.refresh_keys().await?; + Ok(cache) + } + + /// Fetch the JWKS and update the cached keys. + async fn refresh_keys(&self) -> Result<(), String> { + debug!(uri = %self.jwks_uri, "Refreshing JWKS keys"); + + let jwk_set: JwkSet = self + .http + .get(&self.jwks_uri) + .send() + .await + .map_err(|e| format!("JWKS fetch failed: {e}"))? + .json() + .await + .map_err(|e| format!("JWKS parse failed: {e}"))?; + + let mut new_keys = HashMap::new(); + for key in &jwk_set.keys { + if key.kty != "RSA" { + continue; + } + let Some(ref kid) = key.kid else { + continue; + }; + match DecodingKey::from_rsa_components(&key.n, &key.e) { + Ok(dk) => { + new_keys.insert(kid.clone(), dk); + } + Err(e) => { + warn!(kid = %kid, error = %e, "Failed to parse JWK"); + } + } + } + + info!(count = new_keys.len(), "JWKS keys loaded"); + *self.keys.write().await = new_keys; + *self.last_refresh.write().await = Instant::now(); + Ok(()) + } + + /// Refresh keys if the TTL has elapsed. + /// + /// Holds the refresh mutex so concurrent callers coalesce into a single + /// HTTP fetch. The second caller will re-check the TTL after acquiring + /// the lock and find it fresh. + async fn refresh_if_stale(&self) -> Result<(), String> { + let last = *self.last_refresh.read().await; + if last.elapsed() <= self.ttl { + return Ok(()); + } + let _guard = self.refresh_mutex.lock().await; + // Re-check after acquiring the lock — another task may have refreshed. + let last = *self.last_refresh.read().await; + if last.elapsed() <= self.ttl { + return Ok(()); + } + self.refresh_keys().await + } + + /// Refresh keys unconditionally, coalescing concurrent callers. + async fn refresh_keys_coalesced(&self) -> Result<(), String> { + let _guard = self.refresh_mutex.lock().await; + self.refresh_keys().await + } + + /// Validate a JWT and return an `Identity`. + /// + /// This is the authentication step — it verifies the caller's identity + /// but does not check authorization (that's `authz::AuthzPolicy::check`). + pub async fn validate_token(&self, token: &str) -> Result { + self.refresh_if_stale().await.map_err(|e| { + warn!(error = %e, "JWKS refresh failed"); + Status::internal("OIDC key refresh failed") + })?; + + // Decode the header to find the key ID. + let header = decode_header(token).map_err(|e| { + debug!(error = %e, "Failed to decode JWT header"); + Status::unauthenticated("invalid token") + })?; + + let kid = header.kid.ok_or_else(|| { + debug!("JWT has no kid in header"); + Status::unauthenticated("invalid token: missing kid") + })?; + + // Look up the key in cache. + let keys = self.keys.read().await; + let decoding_key = if let Some(k) = keys.get(&kid) { + k.clone() + } else { + // Key not found -- try refreshing once (key rotation). + drop(keys); + self.refresh_keys_coalesced().await.map_err(|e| { + warn!(error = %e, "JWKS refresh on kid miss failed"); + Status::internal("OIDC key refresh failed") + })?; + let keys = self.keys.read().await; + keys.get(&kid).cloned().ok_or_else(|| { + debug!(kid = %kid, "JWT kid not found in JWKS"); + Status::unauthenticated("invalid token: unknown signing key") + })? + }; + + // Validate the JWT. + let mut validation = Validation::new(Algorithm::RS256); + validation.set_issuer(&[&self.config.issuer]); + validation.set_audience(&[&self.config.audience]); + + let token_data = decode::(token, &decoding_key, &validation).map_err(|e| { + debug!(error = %e, "JWT validation failed"); + Status::unauthenticated(format!("invalid token: {e}")) + })?; + + let mut claims = token_data.claims; + claims.extract_roles(&self.config.roles_claim); + + let scopes = if self.config.scopes_claim.is_empty() { + vec![] + } else { + claims.extract_scopes(&self.config.scopes_claim) + }; + + Ok(Identity { + subject: claims.sub, + display_name: claims.preferred_username, + roles: claims.roles, + scopes, + provider: IdentityProvider::Oidc, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn health_is_unauthenticated() { + assert!(is_unauthenticated_method("/openshell.v1.OpenShell/Health")); + } + + #[test] + fn sandbox_operations_require_auth() { + assert!(!is_unauthenticated_method( + "/openshell.v1.OpenShell/CreateSandbox" + )); + assert!(!is_sandbox_secret_method( + "/openshell.v1.OpenShell/CreateSandbox" + )); + } + + #[test] + fn reflection_is_unauthenticated() { + assert!(is_unauthenticated_method( + "/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo" + )); + assert!(is_unauthenticated_method( + "/grpc.reflection.v1.ServerReflection/ServerReflectionInfo" + )); + } + + #[test] + fn grpc_health_is_unauthenticated() { + assert!(is_unauthenticated_method("/grpc.health.v1.Health/Check")); + } + + #[test] + fn sandbox_rpcs_use_sandbox_secret() { + assert!(is_sandbox_secret_method( + "/openshell.sandbox.v1.SandboxService/GetSandboxConfig" + )); + assert!(is_sandbox_secret_method( + "/openshell.v1.OpenShell/GetSandboxProviderEnvironment" + )); + assert!(is_sandbox_secret_method( + "/openshell.v1.OpenShell/ReportPolicyStatus" + )); + assert!(is_sandbox_secret_method( + "/openshell.v1.OpenShell/PushSandboxLogs" + )); + assert!(is_sandbox_secret_method( + "/openshell.v1.OpenShell/SubmitPolicyAnalysis" + )); + assert!(is_sandbox_secret_method( + "/openshell.inference.v1.Inference/GetInferenceBundle" + )); + } + + #[test] + fn openshell_get_sandbox_config_is_dual_auth() { + assert!(!is_sandbox_secret_method( + "/openshell.v1.OpenShell/GetSandboxConfig" + )); + assert!(is_dual_auth_method( + "/openshell.v1.OpenShell/GetSandboxConfig" + )); + } + + #[test] + fn sandbox_secret_validation() { + let mut headers = http::HeaderMap::new(); + headers.insert("x-sandbox-secret", "test-secret".parse().unwrap()); + assert!(validate_sandbox_secret(&headers, "test-secret").is_ok()); + assert!(validate_sandbox_secret(&headers, "wrong-secret").is_err()); + } + + #[test] + fn sandbox_secret_missing_header() { + let headers = http::HeaderMap::new(); + assert!(validate_sandbox_secret(&headers, "test-secret").is_err()); + } + + #[test] + fn extract_roles_keycloak_path() { + let json = serde_json::json!({ + "sub": "user1", + "realm_access": { "roles": ["openshell-user", "openshell-admin"] } + }); + let mut claims: OidcClaims = serde_json::from_value(json).unwrap(); + claims.extract_roles("realm_access.roles"); + assert_eq!(claims.roles, vec!["openshell-user", "openshell-admin"]); + } + + #[test] + fn extract_roles_flat_path() { + // Entra ID / Okta style: roles at top level + let json = serde_json::json!({ + "sub": "user1", + "roles": ["OpenShell.Admin", "OpenShell.User"] + }); + let mut claims: OidcClaims = serde_json::from_value(json).unwrap(); + claims.extract_roles("roles"); + assert_eq!(claims.roles, vec!["OpenShell.Admin", "OpenShell.User"]); + } + + #[test] + fn extract_roles_groups_path() { + // Okta style: groups claim + let json = serde_json::json!({ + "sub": "user1", + "groups": ["everyone", "openshell-admin"] + }); + let mut claims: OidcClaims = serde_json::from_value(json).unwrap(); + claims.extract_roles("groups"); + assert_eq!(claims.roles, vec!["everyone", "openshell-admin"]); + } + + #[test] + fn extract_roles_missing_claim() { + let json = serde_json::json!({ "sub": "user1" }); + let mut claims: OidcClaims = serde_json::from_value(json).unwrap(); + claims.extract_roles("realm_access.roles"); + assert!(claims.roles.is_empty()); + } + + #[test] + fn extract_scopes_space_delimited() { + let json = serde_json::json!({ + "sub": "user1", + "scope": "openid sandbox:read sandbox:write" + }); + let claims: OidcClaims = serde_json::from_value(json).unwrap(); + let scopes = claims.extract_scopes("scope"); + assert_eq!(scopes, vec!["sandbox:read", "sandbox:write"]); + } + + #[test] + fn extract_scopes_json_array() { + let json = serde_json::json!({ + "sub": "user1", + "scp": ["sandbox:read", "provider:read"] + }); + let claims: OidcClaims = serde_json::from_value(json).unwrap(); + let scopes = claims.extract_scopes("scp"); + assert_eq!(scopes, vec!["sandbox:read", "provider:read"]); + } + + #[test] + fn extract_scopes_filters_standard_oidc_scopes() { + let json = serde_json::json!({ + "sub": "user1", + "scope": "openid profile email sandbox:read offline_access" + }); + let claims: OidcClaims = serde_json::from_value(json).unwrap(); + let scopes = claims.extract_scopes("scope"); + assert_eq!(scopes, vec!["sandbox:read"]); + } + + #[test] + fn extract_scopes_missing_claim() { + let json = serde_json::json!({ "sub": "user1" }); + let claims: OidcClaims = serde_json::from_value(json).unwrap(); + let scopes = claims.extract_scopes("scope"); + assert!(scopes.is_empty()); + } + + #[test] + fn extract_scopes_openid_only_yields_empty() { + let json = serde_json::json!({ + "sub": "user1", + "scope": "openid" + }); + let claims: OidcClaims = serde_json::from_value(json).unwrap(); + let scopes = claims.extract_scopes("scope"); + assert!(scopes.is_empty()); + } +} diff --git a/crates/openshell-server/src/cli.rs b/crates/openshell-server/src/cli.rs new file mode 100644 index 0000000000..6eb1ab2db8 --- /dev/null +++ b/crates/openshell-server/src/cli.rs @@ -0,0 +1,544 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Shared CLI entrypoint for the gateway binaries. + +use clap::{Command, CommandFactory, FromArgMatches, Parser}; +use miette::{IntoDiagnostic, Result}; +use openshell_core::ComputeDriverKind; +use openshell_core::config::{ + DEFAULT_DOCKER_NETWORK_NAME, DEFAULT_SERVER_PORT, DEFAULT_SSH_HANDSHAKE_SKEW_SECS, + DEFAULT_SSH_PORT, +}; +use std::net::{IpAddr, SocketAddr}; +use std::path::PathBuf; +use tracing::info; +use tracing_subscriber::EnvFilter; + +use crate::compute::{DockerComputeConfig, VmComputeConfig}; +use crate::{run_server, tracing_bus::TracingLogBus}; + +/// `OpenShell` gateway process - gRPC and HTTP server with protocol multiplexing. +#[derive(Parser, Debug)] +#[command(version = openshell_core::VERSION)] +#[command(about = "OpenShell gRPC/HTTP server", long_about = None)] +struct Args { + /// IP address to bind the server, health, and metrics listeners to. + #[arg(long, default_value = "127.0.0.1", env = "OPENSHELL_BIND_ADDRESS")] + bind_address: IpAddr, + + /// Port to bind the server to. + #[arg(long, default_value_t = DEFAULT_SERVER_PORT, env = "OPENSHELL_SERVER_PORT")] + port: u16, + + /// Port for unauthenticated health endpoints (healthz, readyz). + /// Set to 0 to disable the dedicated health listener. + #[arg(long, default_value_t = 0, env = "OPENSHELL_HEALTH_PORT")] + health_port: u16, + + /// Port for the Prometheus metrics endpoint (/metrics). + /// Set to 0 to disable the dedicated metrics listener. + #[arg(long, default_value_t = 0, env = "OPENSHELL_METRICS_PORT")] + metrics_port: u16, + + /// Log level (trace, debug, info, warn, error). + #[arg(long, default_value = "info", env = "OPENSHELL_LOG_LEVEL")] + log_level: String, + + /// Path to TLS certificate file (required unless --disable-tls). + #[arg(long, env = "OPENSHELL_TLS_CERT")] + tls_cert: Option, + + /// Path to TLS private key file (required unless --disable-tls). + #[arg(long, env = "OPENSHELL_TLS_KEY")] + tls_key: Option, + + /// Path to CA certificate for client certificate verification (mTLS). + #[arg(long, env = "OPENSHELL_TLS_CLIENT_CA")] + tls_client_ca: Option, + + /// Database URL for persistence. + #[arg(long, env = "OPENSHELL_DB_URL", required = true)] + db_url: String, + + /// Compute drivers configured for this gateway. + /// + /// Accepts a comma-delimited list such as `kubernetes` or + /// `kubernetes,podman`. The configuration format is future-proofed for + /// multiple drivers, but the gateway currently requires exactly one. + /// When unset, the gateway auto-detects the driver based on the runtime + /// environment (Kubernetes → Podman → Docker). VM is never auto-detected + /// and requires explicit configuration. + #[arg( + long, + alias = "driver", + env = "OPENSHELL_DRIVERS", + value_delimiter = ',', + value_parser = parse_compute_driver + )] + drivers: Vec, + + /// Kubernetes namespace for sandboxes. + #[arg(long, env = "OPENSHELL_SANDBOX_NAMESPACE", default_value = "default")] + sandbox_namespace: String, + + /// Default container image for sandboxes. + #[arg(long, env = "OPENSHELL_SANDBOX_IMAGE")] + sandbox_image: Option, + + /// Kubernetes `imagePullPolicy` for sandbox pods (Always, `IfNotPresent`, Never). + #[arg(long, env = "OPENSHELL_SANDBOX_IMAGE_PULL_POLICY")] + sandbox_image_pull_policy: Option, + + /// gRPC endpoint for sandboxes to callback to `OpenShell`. + /// This should be reachable from within the Kubernetes cluster. + #[arg(long, env = "OPENSHELL_GRPC_ENDPOINT")] + grpc_endpoint: Option, + + /// Public host for the SSH gateway. + #[arg(long, env = "OPENSHELL_SSH_GATEWAY_HOST", default_value = "127.0.0.1")] + ssh_gateway_host: String, + + /// Public port for the SSH gateway. + #[arg(long, env = "OPENSHELL_SSH_GATEWAY_PORT", default_value_t = DEFAULT_SERVER_PORT)] + ssh_gateway_port: u16, + + /// HTTP path for SSH CONNECT/upgrade. + #[arg( + long, + env = "OPENSHELL_SSH_CONNECT_PATH", + default_value = "/connect/ssh" + )] + ssh_connect_path: String, + + /// SSH port inside sandbox pods. + #[arg(long, env = "OPENSHELL_SANDBOX_SSH_PORT", default_value_t = DEFAULT_SSH_PORT)] + sandbox_ssh_port: u16, + /// Shared secret for gateway-to-sandbox SSH handshake. + #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SECRET")] + ssh_handshake_secret: Option, + + /// Allowed clock skew in seconds for SSH handshake. + #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SKEW_SECS", default_value_t = DEFAULT_SSH_HANDSHAKE_SKEW_SECS)] + ssh_handshake_skew_secs: u64, + + /// Kubernetes secret name containing client TLS materials for sandbox pods. + #[arg(long, env = "OPENSHELL_CLIENT_TLS_SECRET_NAME")] + client_tls_secret_name: Option, + + /// Host gateway IP for sandbox pod hostAliases. + /// When set, sandbox pods get hostAliases entries mapping + /// host.docker.internal and host.openshell.internal to this IP. + #[arg(long, env = "OPENSHELL_HOST_GATEWAY_IP")] + host_gateway_ip: Option, + + /// Working directory for VM driver sandbox state. + #[arg( + long, + env = "OPENSHELL_VM_DRIVER_STATE_DIR", + default_value_os_t = VmComputeConfig::default_state_dir() + )] + vm_driver_state_dir: PathBuf, + + /// Directory searched for compute-driver binaries (e.g. + /// `openshell-driver-vm`) when an explicit binary override isn't + /// configured. When unset, the gateway searches + /// `$HOME/.local/libexec/openshell`, `/usr/libexec/openshell`, + /// `/usr/local/libexec/openshell`, `/usr/local/libexec`, then a sibling + /// of the gateway binary. + #[arg(long, env = "OPENSHELL_DRIVER_DIR")] + driver_dir: Option, + + /// libkrun log level used by the VM helper. + #[arg( + long, + env = "OPENSHELL_VM_KRUN_LOG_LEVEL", + default_value_t = VmComputeConfig::default_krun_log_level() + )] + vm_krun_log_level: u32, + + /// Default vCPU count for VM sandboxes. + #[arg( + long, + env = "OPENSHELL_VM_DRIVER_VCPUS", + default_value_t = VmComputeConfig::default_vcpus() + )] + vm_vcpus: u8, + + /// Default memory allocation for VM sandboxes, in MiB. + #[arg( + long, + env = "OPENSHELL_VM_DRIVER_MEM_MIB", + default_value_t = VmComputeConfig::default_mem_mib() + )] + vm_mem_mib: u32, + + /// CA certificate installed into VM sandboxes for gateway mTLS. + #[arg(long, env = "OPENSHELL_VM_TLS_CA")] + vm_tls_ca: Option, + + /// Client certificate installed into VM sandboxes for gateway mTLS. + #[arg(long, env = "OPENSHELL_VM_TLS_CERT")] + vm_tls_cert: Option, + + /// Client private key installed into VM sandboxes for gateway mTLS. + #[arg(long, env = "OPENSHELL_VM_TLS_KEY")] + vm_tls_key: Option, + + /// Linux `openshell-sandbox` binary bind-mounted into Docker sandboxes. + /// + /// When unset the gateway falls back to (in order) a sibling + /// `openshell-sandbox` next to the gateway binary, a local cargo build, + /// or extracting the binary from `--docker-supervisor-image`. + #[arg(long, env = "OPENSHELL_DOCKER_SUPERVISOR_BIN")] + docker_supervisor_bin: Option, + + /// Image the Docker driver pulls to extract the Linux + /// `openshell-sandbox` binary when no explicit `--docker-supervisor-bin` + /// override or local build is available. Defaults to + /// `ghcr.io/nvidia/openshell/supervisor:`. + #[arg(long, env = "OPENSHELL_DOCKER_SUPERVISOR_IMAGE")] + docker_supervisor_image: Option, + + /// CA certificate bind-mounted into Docker sandboxes for gateway mTLS. + #[arg(long, env = "OPENSHELL_DOCKER_TLS_CA")] + docker_tls_ca: Option, + + /// Client certificate bind-mounted into Docker sandboxes for gateway mTLS. + #[arg(long, env = "OPENSHELL_DOCKER_TLS_CERT")] + docker_tls_cert: Option, + + /// Client private key bind-mounted into Docker sandboxes for gateway mTLS. + #[arg(long, env = "OPENSHELL_DOCKER_TLS_KEY")] + docker_tls_key: Option, + + /// Docker bridge network used for sandbox containers. + #[arg( + long, + env = "OPENSHELL_DOCKER_NETWORK_NAME", + default_value = DEFAULT_DOCKER_NETWORK_NAME + )] + docker_network_name: String, + + /// Disable TLS entirely — listen on plaintext HTTP. + /// Use this when the gateway sits behind a reverse proxy or tunnel + /// (e.g. Cloudflare Tunnel) that terminates TLS at the edge. + #[arg(long, env = "OPENSHELL_DISABLE_TLS")] + disable_tls: bool, + + /// Disable gateway authentication (mTLS client certificate requirement). + /// When set, the TLS handshake accepts connections without a client + /// certificate. Ignored when --disable-tls is set. + #[arg(long, env = "OPENSHELL_DISABLE_GATEWAY_AUTH")] + disable_gateway_auth: bool, + + /// OIDC issuer URL for JWT-based authentication. + /// When set, the server validates `authorization: Bearer` tokens on gRPC + /// requests against the issuer's JWKS endpoint. + #[arg(long, env = "OPENSHELL_OIDC_ISSUER")] + oidc_issuer: Option, + + /// Expected OIDC audience claim (typically the client ID). + #[arg(long, env = "OPENSHELL_OIDC_AUDIENCE", default_value = "openshell-cli")] + oidc_audience: String, + + /// JWKS key cache TTL in seconds. + #[arg(long, env = "OPENSHELL_OIDC_JWKS_TTL", default_value_t = 3600)] + oidc_jwks_ttl: u64, + + /// Dot-separated path to the roles array in the JWT claims. + /// Keycloak: `realm_access.roles` (default). Entra ID: "roles". Okta: "groups". + #[arg( + long, + env = "OPENSHELL_OIDC_ROLES_CLAIM", + default_value = "realm_access.roles" + )] + oidc_roles_claim: String, + + /// Role name that grants admin access. + #[arg( + long, + env = "OPENSHELL_OIDC_ADMIN_ROLE", + default_value = "openshell-admin" + )] + oidc_admin_role: String, + + /// Role name that grants standard user access. + #[arg( + long, + env = "OPENSHELL_OIDC_USER_ROLE", + default_value = "openshell-user" + )] + oidc_user_role: String, + + /// Dot-separated path to the scopes value in the JWT claims. + /// When set, the server enforces scope-based permissions on top of roles. + /// Keycloak: "scope". Okta: "scp". Leave empty to disable scope enforcement. + #[arg(long, env = "OPENSHELL_OIDC_SCOPES_CLAIM", default_value = "")] + oidc_scopes_claim: String, +} + +pub fn command() -> Command { + Args::command() + .name("openshell-gateway") + .bin_name("openshell-gateway") +} + +pub async fn run_cli() -> Result<()> { + rustls::crypto::ring::default_provider() + .install_default() + .map_err(|e| miette::miette!("failed to install rustls crypto provider: {e:?}"))?; + + let args = Args::from_arg_matches(&command().get_matches()).expect("clap validated args"); + + Box::pin(run_from_args(args)).await +} + +async fn run_from_args(args: Args) -> Result<()> { + let tracing_log_bus = TracingLogBus::new(); + tracing_log_bus.install_subscriber( + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)), + ); + + let bind = SocketAddr::new(args.bind_address, args.port); + + let tls = if args.disable_tls { + None + } else { + let cert_path = args.tls_cert.ok_or_else(|| { + miette::miette!( + "--tls-cert is required when TLS is enabled (use --disable-tls to skip)" + ) + })?; + let key_path = args.tls_key.ok_or_else(|| { + miette::miette!("--tls-key is required when TLS is enabled (use --disable-tls to skip)") + })?; + let client_ca_path = args.tls_client_ca.ok_or_else(|| { + miette::miette!( + "--tls-client-ca is required when TLS is enabled (use --disable-tls to skip)" + ) + })?; + Some(openshell_core::TlsConfig { + cert_path, + key_path, + client_ca_path, + allow_unauthenticated: args.disable_gateway_auth, + }) + }; + + let mut config = openshell_core::Config::new(tls) + .with_bind_address(bind) + .with_log_level(&args.log_level); + + if args.health_port != 0 { + if args.port == args.health_port { + return Err(miette::miette!( + "--port and --health-port must be different (both set to {})", + args.port + )); + } + let health_bind = SocketAddr::new(args.bind_address, args.health_port); + config = config.with_health_bind_address(health_bind); + } + + if args.metrics_port != 0 { + if args.port == args.metrics_port { + return Err(miette::miette!( + "--port and --metrics-port must be different (both set to {})", + args.port + )); + } + if args.health_port != 0 && args.health_port == args.metrics_port { + return Err(miette::miette!( + "--health-port and --metrics-port must be different (both set to {})", + args.health_port + )); + } + let metrics_bind = SocketAddr::new(args.bind_address, args.metrics_port); + config = config.with_metrics_bind_address(metrics_bind); + } + + config = config + .with_database_url(args.db_url) + .with_compute_drivers(args.drivers) + .with_sandbox_namespace(args.sandbox_namespace) + .with_ssh_gateway_host(args.ssh_gateway_host) + .with_ssh_gateway_port(args.ssh_gateway_port) + .with_ssh_connect_path(args.ssh_connect_path) + .with_sandbox_ssh_port(args.sandbox_ssh_port) + .with_ssh_handshake_skew_secs(args.ssh_handshake_skew_secs); + + if let Some(image) = args.sandbox_image { + config = config.with_sandbox_image(image); + } + + if let Some(policy) = args.sandbox_image_pull_policy { + config = config.with_sandbox_image_pull_policy(policy); + } + + if let Some(endpoint) = args.grpc_endpoint { + config = config.with_grpc_endpoint(endpoint); + } + + if let Some(secret) = args.ssh_handshake_secret { + config = config.with_ssh_handshake_secret(secret); + } + + if let Some(name) = args.client_tls_secret_name { + config = config.with_client_tls_secret_name(name); + } + + if let Some(ip) = args.host_gateway_ip { + config = config.with_host_gateway_ip(ip); + } + + if let Some(issuer) = args.oidc_issuer { + config = config.with_oidc(openshell_core::OidcConfig { + issuer, + audience: args.oidc_audience, + jwks_ttl_secs: args.oidc_jwks_ttl, + roles_claim: args.oidc_roles_claim, + admin_role: args.oidc_admin_role, + user_role: args.oidc_user_role, + scopes_claim: args.oidc_scopes_claim, + }); + } + + let vm_config = VmComputeConfig { + state_dir: args.vm_driver_state_dir, + driver_dir: args.driver_dir, + default_image: config.sandbox_image.clone(), + krun_log_level: args.vm_krun_log_level, + vcpus: args.vm_vcpus, + mem_mib: args.vm_mem_mib, + guest_tls_ca: args.vm_tls_ca, + guest_tls_cert: args.vm_tls_cert, + guest_tls_key: args.vm_tls_key, + }; + + let docker_config = DockerComputeConfig { + supervisor_bin: args.docker_supervisor_bin, + supervisor_image: args.docker_supervisor_image, + guest_tls_ca: args.docker_tls_ca, + guest_tls_cert: args.docker_tls_cert, + guest_tls_key: args.docker_tls_key, + network_name: args.docker_network_name, + }; + + if args.disable_tls { + info!("TLS disabled — listening on plaintext HTTP"); + } else if args.disable_gateway_auth { + info!("Gateway auth disabled — accepting connections without client certificates"); + } + + info!(bind = %config.bind_address, "Starting OpenShell server"); + + run_server(config, vm_config, docker_config, tracing_log_bus) + .await + .into_diagnostic() +} + +fn parse_compute_driver(value: &str) -> std::result::Result { + value.parse() +} + +#[cfg(test)] +mod tests { + use super::{Args, command}; + use clap::Parser; + use std::net::{IpAddr, Ipv4Addr}; + use std::sync::{LazyLock, Mutex}; + + static ENV_LOCK: LazyLock> = LazyLock::new(|| Mutex::new(())); + + struct EnvVarGuard { + key: &'static str, + original: Option, + } + + impl EnvVarGuard { + #[allow(unsafe_code)] + fn set(key: &'static str, value: &str) -> Self { + let original = std::env::var(key).ok(); + // SAFETY: tests serialize environment mutation with ENV_LOCK. + unsafe { std::env::set_var(key, value) }; + Self { key, original } + } + + #[allow(unsafe_code)] + fn remove(key: &'static str) -> Self { + let original = std::env::var(key).ok(); + // SAFETY: tests serialize environment mutation with ENV_LOCK. + unsafe { std::env::remove_var(key) }; + Self { key, original } + } + } + + impl Drop for EnvVarGuard { + #[allow(unsafe_code)] + fn drop(&mut self) { + match self.original.as_deref() { + // SAFETY: tests serialize environment mutation with ENV_LOCK. + Some(value) => unsafe { std::env::set_var(self.key, value) }, + // SAFETY: tests serialize environment mutation with ENV_LOCK. + None => unsafe { std::env::remove_var(self.key) }, + } + } + } + + #[test] + fn command_uses_gateway_binary_name() { + let mut help = Vec::new(); + command().write_long_help(&mut help).unwrap(); + let help = String::from_utf8(help).unwrap(); + assert!(help.contains("openshell-gateway")); + } + + #[test] + fn command_exposes_version() { + let cmd = command(); + let version = cmd.get_version().unwrap(); + assert_eq!(version.to_string(), openshell_core::VERSION); + } + + #[test] + fn command_defaults_bind_address_to_loopback() { + let _lock = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let _guard = EnvVarGuard::remove("OPENSHELL_BIND_ADDRESS"); + let args = + Args::try_parse_from(["openshell-gateway", "--db-url", "sqlite::memory:"]).unwrap(); + assert_eq!(args.bind_address, IpAddr::V4(Ipv4Addr::LOCALHOST)); + } + + #[test] + fn command_parses_bind_address() { + let _lock = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let _guard = EnvVarGuard::remove("OPENSHELL_BIND_ADDRESS"); + let args = Args::try_parse_from([ + "openshell-gateway", + "--db-url", + "sqlite::memory:", + "--bind-address", + "127.0.0.1", + ]) + .unwrap(); + assert_eq!(args.bind_address, IpAddr::V4(Ipv4Addr::LOCALHOST)); + } + + #[test] + fn command_reads_bind_address_from_env() { + let _lock = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let _guard = EnvVarGuard::set("OPENSHELL_BIND_ADDRESS", "0.0.0.0"); + + let args = Args::try_parse_from(["openshell-gateway", "--db-url", "sqlite::memory:"]) + .expect("env should provide bind address"); + + assert_eq!(args.bind_address, IpAddr::V4(Ipv4Addr::UNSPECIFIED)); + } +} diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 846782c65e..2d6351637d 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -3,34 +3,45 @@ //! Gateway-owned compute orchestration over a pluggable compute backend. -use crate::grpc::policy::{SANDBOX_SETTINGS_OBJECT_TYPE, sandbox_settings_id}; +pub mod vm; + +pub use openshell_driver_docker::DockerComputeConfig; +pub use vm::VmComputeConfig; + +use crate::grpc::policy::SANDBOX_SETTINGS_OBJECT_TYPE; use crate::persistence::{ObjectId, ObjectName, ObjectRecord, ObjectType, Store}; use crate::sandbox_index::SandboxIndex; use crate::sandbox_watch::SandboxWatchBus; +use crate::supervisor_session::SupervisorSessionRegistry; use crate::tracing_bus::TracingLogBus; use futures::{Stream, StreamExt}; use openshell_core::proto::compute::v1::{ CreateSandboxRequest, DeleteSandboxRequest, DriverCondition, DriverPlatformEvent, DriverResourceRequirements, DriverSandbox, DriverSandboxSpec, DriverSandboxStatus, DriverSandboxTemplate, GetCapabilitiesRequest, GetSandboxRequest, ListSandboxesRequest, - ResolveSandboxEndpointRequest, ResolveSandboxEndpointResponse, ValidateSandboxCreateRequest, - WatchSandboxesEvent, WatchSandboxesRequest, compute_driver_server::ComputeDriver, - sandbox_endpoint, watch_sandboxes_event, + ValidateSandboxCreateRequest, WatchSandboxesEvent, WatchSandboxesRequest, + compute_driver_client::ComputeDriverClient, compute_driver_server::ComputeDriver, + watch_sandboxes_event, }; use openshell_core::proto::{ PlatformEvent, Sandbox, SandboxCondition, SandboxPhase, SandboxSpec, SandboxStatus, SandboxTemplate, SshSession, }; +use openshell_driver_docker::DockerComputeDriver; use openshell_driver_kubernetes::{ ComputeDriverService, KubernetesComputeConfig, KubernetesComputeDriver, }; +use openshell_driver_podman::{ + ComputeDriverService as PodmanDriverService, PodmanComputeConfig, PodmanComputeDriver, +}; use prost::Message; use std::fmt; -use std::net::IpAddr; +use std::net::SocketAddr; use std::pin::Pin; use std::sync::Arc; use std::time::Duration; use tokio::sync::Mutex; +use tonic::transport::Channel; use tonic::{Code, Request, Status}; use tracing::{info, warn}; @@ -38,6 +49,44 @@ type DriverWatchStream = Pin + Send + Sync>; +#[tonic::async_trait] +trait ShutdownCleanup: Send + Sync { + async fn cleanup_on_shutdown(&self) -> Result<(), String>; +} + +#[tonic::async_trait] +impl ShutdownCleanup for DockerComputeDriver { + async fn cleanup_on_shutdown(&self) -> Result<(), String> { + let stopped = self + .stop_managed_containers_on_shutdown() + .await + .map_err(|err| err.to_string())?; + info!( + stopped_containers = stopped, + "Stopped Docker sandbox containers during gateway shutdown" + ); + Ok(()) + } +} + +/// Resume a single sandbox whose store record indicates it should be +/// running. Implemented by drivers (currently only Docker) where compute +/// resources do not auto-restart with the gateway. Returns `Ok(true)` if +/// the backend resource was found and resumed (or was already running), +/// `Ok(false)` if no backend resource exists. +#[tonic::async_trait] +trait StartupResume: Send + Sync { + async fn resume_sandbox(&self, sandbox_id: &str, sandbox_name: &str) -> Result; +} + +#[tonic::async_trait] +impl StartupResume for DockerComputeDriver { + async fn resume_sandbox(&self, sandbox_id: &str, sandbox_name: &str) -> Result { + Self::resume_sandbox(self, sandbox_id, sandbox_name) + .await + .map_err(|err| err.to_string()) + } +} /// Interval between store-vs-backend reconciliation sweeps. const RECONCILE_INTERVAL: Duration = Duration::from_secs(60); @@ -45,31 +94,142 @@ const RECONCILE_INTERVAL: Duration = Duration::from_secs(60); /// corresponding backend resource before it is considered orphaned. const ORPHAN_GRACE_PERIOD: Duration = Duration::from_secs(300); -#[derive(Debug, thiserror::Error)] -pub enum ComputeError { - #[error("sandbox already exists")] - AlreadyExists, - #[error("{0}")] - Precondition(String), - #[error("{0}")] - Message(String), -} +// Re-export the shared error type under the name used by this module. +pub use openshell_core::ComputeDriverError as ComputeError; #[derive(Debug)] -pub enum ResolvedEndpoint { - Ip(IpAddr, u16), - Host(String, u16), +pub struct ManagedDriverProcess { + child: std::sync::Mutex>, + socket_path: std::path::PathBuf, +} + +impl ManagedDriverProcess { + pub(crate) fn new(child: tokio::process::Child, socket_path: std::path::PathBuf) -> Self { + Self { + child: std::sync::Mutex::new(Some(child)), + socket_path, + } + } +} + +impl Drop for ManagedDriverProcess { + fn drop(&mut self) { + if let Ok(mut child) = self.child.lock() { + let _ = child.take(); + } + let _ = std::fs::remove_file(&self.socket_path); + } +} + +#[derive(Debug, Clone)] +struct RemoteComputeDriver { + channel: Channel, +} + +impl RemoteComputeDriver { + fn new(channel: Channel) -> Self { + Self { channel } + } + + fn client(&self) -> ComputeDriverClient { + ComputeDriverClient::new(self.channel.clone()) + } +} + +#[tonic::async_trait] +impl ComputeDriver for RemoteComputeDriver { + type WatchSandboxesStream = DriverWatchStream; + + async fn get_capabilities( + &self, + request: Request, + ) -> Result, Status> + { + let mut client = self.client(); + client.get_capabilities(request).await + } + + async fn validate_sandbox_create( + &self, + request: Request, + ) -> Result< + tonic::Response, + Status, + > { + let mut client = self.client(); + client.validate_sandbox_create(request).await + } + + async fn get_sandbox( + &self, + request: Request, + ) -> Result, Status> + { + let mut client = self.client(); + client.get_sandbox(request).await + } + + async fn list_sandboxes( + &self, + request: Request, + ) -> Result, Status> + { + let mut client = self.client(); + client.list_sandboxes(request).await + } + + async fn create_sandbox( + &self, + request: Request, + ) -> Result, Status> + { + let mut client = self.client(); + client.create_sandbox(request).await + } + + async fn stop_sandbox( + &self, + request: Request, + ) -> Result, Status> + { + let mut client = self.client(); + client.stop_sandbox(request).await + } + + async fn delete_sandbox( + &self, + request: Request, + ) -> Result, Status> + { + let mut client = self.client(); + client.delete_sandbox(request).await + } + + async fn watch_sandboxes( + &self, + request: Request, + ) -> Result, Status> { + let mut client = self.client(); + let response = client.watch_sandboxes(request).await?; + let stream = response.into_inner(); + Ok(tonic::Response::new(Box::pin(stream))) + } } #[derive(Clone)] pub struct ComputeRuntime { driver: SharedComputeDriver, + shutdown_cleanup: Option>, + startup_resume: Option>, + _driver_process: Option>, default_image: String, store: Arc, sandbox_index: SandboxIndex, sandbox_watch_bus: SandboxWatchBus, tracing_log_bus: TracingLogBus, + supervisor_sessions: Arc, sync_lock: Arc>, + gateway_bind_addresses: Vec, } impl fmt::Debug for ComputeRuntime { @@ -79,17 +239,20 @@ impl fmt::Debug for ComputeRuntime { } impl ComputeRuntime { - pub async fn new_kubernetes( - config: KubernetesComputeConfig, + #[allow(clippy::too_many_arguments)] + async fn from_driver( + driver: SharedComputeDriver, + shutdown_cleanup: Option>, + startup_resume: Option>, + driver_process: Option>, store: Arc, sandbox_index: SandboxIndex, sandbox_watch_bus: SandboxWatchBus, tracing_log_bus: TracingLogBus, + supervisor_sessions: Arc, + _allows_loopback_endpoints: bool, + gateway_bind_addresses: Vec, ) -> Result { - let driver = KubernetesComputeDriver::new(config) - .await - .map_err(|err| ComputeError::Message(err.to_string()))?; - let driver: SharedComputeDriver = Arc::new(ComputeDriverService::new(driver)); let default_image = driver .get_capabilities(Request::new(GetCapabilitiesRequest {})) .await @@ -98,20 +261,146 @@ impl ComputeRuntime { .default_image; Ok(Self { driver, + shutdown_cleanup, + startup_resume, + _driver_process: driver_process, default_image, store, sandbox_index, sandbox_watch_bus, tracing_log_bus, + supervisor_sessions, sync_lock: Arc::new(Mutex::new(())), + gateway_bind_addresses, }) } + pub async fn new_docker( + config: openshell_core::Config, + docker_config: DockerComputeConfig, + store: Arc, + sandbox_index: SandboxIndex, + sandbox_watch_bus: SandboxWatchBus, + tracing_log_bus: TracingLogBus, + supervisor_sessions: Arc, + ) -> Result { + let driver = Arc::new( + DockerComputeDriver::new(&config, &docker_config, supervisor_sessions.clone()) + .await + .map_err(|err| ComputeError::Message(err.to_string()))?, + ); + let gateway_bind_addresses = driver.gateway_bind_addresses(); + let shutdown_cleanup: Arc = driver.clone(); + let startup_resume: Arc = driver.clone(); + let driver: SharedComputeDriver = driver; + Self::from_driver( + driver, + Some(shutdown_cleanup), + Some(startup_resume), + None, + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions, + true, + gateway_bind_addresses, + ) + .await + } + + pub async fn new_kubernetes( + config: KubernetesComputeConfig, + store: Arc, + sandbox_index: SandboxIndex, + sandbox_watch_bus: SandboxWatchBus, + tracing_log_bus: TracingLogBus, + supervisor_sessions: Arc, + ) -> Result { + let driver = KubernetesComputeDriver::new(config) + .await + .map_err(|err| ComputeError::Message(err.to_string()))?; + let driver: SharedComputeDriver = Arc::new(ComputeDriverService::new(driver)); + Self::from_driver( + driver, + None, + None, + None, + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions, + false, + Vec::new(), + ) + .await + } + + pub(crate) async fn new_remote_vm( + channel: Channel, + driver_process: Option>, + store: Arc, + sandbox_index: SandboxIndex, + sandbox_watch_bus: SandboxWatchBus, + tracing_log_bus: TracingLogBus, + supervisor_sessions: Arc, + ) -> Result { + let driver: SharedComputeDriver = Arc::new(RemoteComputeDriver::new(channel)); + Self::from_driver( + driver, + None, + None, + driver_process, + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions, + true, + Vec::new(), + ) + .await + } + + pub async fn new_podman( + config: PodmanComputeConfig, + store: Arc, + sandbox_index: SandboxIndex, + sandbox_watch_bus: SandboxWatchBus, + tracing_log_bus: TracingLogBus, + supervisor_sessions: Arc, + ) -> Result { + let driver = PodmanComputeDriver::new(config) + .await + .map_err(|err| ComputeError::Message(err.to_string()))?; + let driver: SharedComputeDriver = Arc::new(PodmanDriverService::new(driver)); + Self::from_driver( + driver, + None, + None, + None, + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions, + true, + Vec::new(), + ) + .await + } + #[must_use] pub fn default_image(&self) -> &str { &self.default_image } + #[must_use] + pub fn gateway_bind_addresses(&self) -> &[SocketAddr] { + &self.gateway_bind_addresses + } + pub async fn validate_sandbox_create(&self, sandbox: &Sandbox) -> Result<(), Status> { let driver_sandbox = driver_sandbox_from_public(sandbox); self.driver @@ -125,13 +414,13 @@ impl ComputeRuntime { pub async fn create_sandbox(&self, sandbox: Sandbox) -> Result { let existing = self .store - .get_message_by_name::(&sandbox.name) + .get_message_by_name::(sandbox.object_name()) .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))?; if existing.is_some() { return Err(Status::already_exists(format!( "sandbox '{}' already exists", - sandbox.name + sandbox.object_name() ))); } @@ -150,22 +439,31 @@ impl ComputeRuntime { .await { Ok(_) => { - self.sandbox_watch_bus.notify(&sandbox.id); + self.sandbox_watch_bus.notify(sandbox.object_id()); Ok(sandbox) } Err(status) if status.code() == Code::AlreadyExists => { - let _ = self.store.delete(Sandbox::object_type(), &sandbox.id).await; - self.sandbox_index.remove_sandbox(&sandbox.id); + let _ = self + .store + .delete(Sandbox::object_type(), sandbox.object_id()) + .await; + self.sandbox_index.remove_sandbox(sandbox.object_id()); Err(Status::already_exists("sandbox already exists")) } Err(status) if status.code() == Code::FailedPrecondition => { - let _ = self.store.delete(Sandbox::object_type(), &sandbox.id).await; - self.sandbox_index.remove_sandbox(&sandbox.id); + let _ = self + .store + .delete(Sandbox::object_type(), sandbox.object_id()) + .await; + self.sandbox_index.remove_sandbox(sandbox.object_id()); Err(Status::failed_precondition(status.message().to_string())) } Err(err) => { - let _ = self.store.delete(Sandbox::object_type(), &sandbox.id).await; - self.sandbox_index.remove_sandbox(&sandbox.id); + let _ = self + .store + .delete(Sandbox::object_type(), sandbox.object_id()) + .await; + self.sandbox_index.remove_sandbox(sandbox.object_id()); Err(Status::internal(format!( "create sandbox failed: {}", err.message() @@ -185,7 +483,7 @@ impl ComputeRuntime { return Err(Status::not_found("sandbox not found")); }; - let id = sandbox.id.clone(); + let id = sandbox.object_id().to_string(); sandbox.phase = SandboxPhase::Deleting as i32; self.store .put_message(&sandbox) @@ -193,36 +491,7 @@ impl ComputeRuntime { .map_err(|e| Status::internal(format!("persist sandbox failed: {e}")))?; self.sandbox_index.update_from_sandbox(&sandbox); self.sandbox_watch_bus.notify(&id); - - if let Ok(records) = self.store.list(SshSession::object_type(), 1000, 0).await { - for record in records { - if let Ok(session) = SshSession::decode(record.payload.as_slice()) - && session.sandbox_id == id - && let Err(e) = self - .store - .delete(SshSession::object_type(), &session.id) - .await - { - warn!( - session_id = %session.id, - error = %e, - "Failed to delete SSH session during sandbox cleanup" - ); - } - } - } - - if let Err(e) = self - .store - .delete(SANDBOX_SETTINGS_OBJECT_TYPE, &sandbox_settings_id(&id)) - .await - { - warn!( - sandbox_id = %id, - error = %e, - "Failed to delete sandbox settings during cleanup" - ); - } + self.cleanup_sandbox_owned_records(&sandbox).await; let driver_sandbox = driver_sandbox_from_public(&sandbox); let deleted = self @@ -243,29 +512,6 @@ impl ComputeRuntime { Ok(deleted) } - pub async fn resolve_sandbox_endpoint( - &self, - sandbox: &Sandbox, - ) -> Result { - let driver_sandbox = driver_sandbox_from_public(sandbox); - self.driver - .resolve_sandbox_endpoint(Request::new(ResolveSandboxEndpointRequest { - sandbox: Some(driver_sandbox), - })) - .await - .map(|response| response.into_inner()) - .map_err(|status| match status.code() { - Code::FailedPrecondition => { - Status::failed_precondition(status.message().to_string()) - } - _ => Status::internal(status.message().to_string()), - }) - .and_then(|response| { - resolved_endpoint_from_response(&response) - .map_err(|err| Status::internal(err.to_string())) - }) - } - pub fn spawn_watchers(&self) { let runtime = Arc::new(self.clone()); let watch_runtime = runtime.clone(); @@ -277,6 +523,142 @@ impl ComputeRuntime { }); } + pub async fn cleanup_on_shutdown(&self) -> Result<(), String> { + let Some(cleanup) = &self.shutdown_cleanup else { + return Ok(()); + }; + cleanup.cleanup_on_shutdown().await + } + + /// Resume sandboxes whose store records say they should be running. + /// Drivers that do not auto-restart compute resources across gateway + /// restarts (currently only Docker) implement `StartupResume`. For + /// each sandbox in the store whose phase is not `Deleting` or + /// `Error`, we ask the driver to resume the underlying resource. If + /// the driver reports that the resource no longer exists or fails to + /// start, the sandbox is moved to the `Error` phase so the failure + /// surfaces in the UI. + /// + /// Should be called once at gateway startup, before watchers spawn, + /// so the watch loop sees the post-resume state on its first poll. + pub async fn resume_persisted_sandboxes(&self) -> Result<(), String> { + let Some(resume) = &self.startup_resume else { + return Ok(()); + }; + + let records = self + .store + .list(Sandbox::object_type(), 1000, 0) + .await + .map_err(|e| e.to_string())?; + + let mut resumed = 0usize; + let mut missing = 0usize; + let mut failed = 0usize; + + for record in records { + let sandbox = match Sandbox::decode(record.payload.as_slice()) { + Ok(sandbox) => sandbox, + Err(err) => { + warn!(error = %err, "Failed to decode sandbox record during startup resume"); + continue; + } + }; + + let phase = SandboxPhase::try_from(sandbox.phase).unwrap_or(SandboxPhase::Unknown); + if !sandbox_phase_should_be_running(phase) { + continue; + } + + match resume + .resume_sandbox(sandbox.object_id(), sandbox.object_name()) + .await + { + Ok(true) => { + info!( + sandbox_id = %sandbox.object_id(), + sandbox_name = %sandbox.object_name(), + ?phase, + "Resumed sandbox during gateway startup" + ); + resumed += 1; + } + Ok(false) => { + // Backend resource is gone but the store still + // remembers the sandbox. Mark Error so the UI + // surfaces the inconsistency; the reconcile loop + // will eventually prune it after the orphan grace + // period. + warn!( + sandbox_id = %sandbox.object_id(), + sandbox_name = %sandbox.object_name(), + "Cannot resume sandbox: backend resource is missing" + ); + self.mark_sandbox_error( + &sandbox, + "BackendResourceMissing", + "Sandbox container disappeared while the gateway was offline", + ) + .await; + missing += 1; + } + Err(err) => { + warn!( + sandbox_id = %sandbox.object_id(), + sandbox_name = %sandbox.object_name(), + error = %err, + "Failed to resume sandbox during gateway startup" + ); + self.mark_sandbox_error( + &sandbox, + "ResumeFailed", + &format!("Failed to resume sandbox during gateway startup: {err}"), + ) + .await; + failed += 1; + } + } + } + + if resumed > 0 || missing > 0 || failed > 0 { + info!( + resumed, + missing_backend = missing, + failed, + "Sandbox resume sweep complete" + ); + } + Ok(()) + } + + async fn mark_sandbox_error(&self, sandbox: &Sandbox, reason: &str, message: &str) { + let _guard = self.sync_lock.lock().await; + let mut updated = sandbox.clone(); + updated.phase = SandboxPhase::Error as i32; + let updated_name = updated.object_name().to_string(); + upsert_ready_condition( + &mut updated.status, + &updated_name, + SandboxCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: reason.to_string(), + message: message.to_string(), + last_transition_time: String::new(), + }, + ); + self.sandbox_index.update_from_sandbox(&updated); + if let Err(err) = self.store.put_message(&updated).await { + warn!( + sandbox_id = %sandbox.object_id(), + error = %err, + "Failed to persist sandbox error state during startup resume" + ); + return; + } + self.sandbox_watch_bus.notify(sandbox.object_id()); + } + async fn watch_loop(self: Arc) { loop { let mut stream = match self @@ -360,7 +742,7 @@ impl ComputeRuntime { } }; - if backend_ids.contains(&sandbox.id) { + if backend_ids.contains(sandbox.object_id()) { continue; } @@ -427,17 +809,31 @@ impl ComputeRuntime { existing.as_ref().and_then(|sandbox| sandbox.spec.as_ref()), ); - let phase = derive_phase(incoming.status.as_ref()); - let mut sandbox = existing.unwrap_or_else(|| Sandbox { - id: incoming.id.clone(), - name: incoming.name.clone(), - namespace: incoming.namespace.clone(), - spec: None, - status: None, - phase: SandboxPhase::Unknown as i32, - ..Default::default() + let session_connected = self.supervisor_sessions.has_session(&incoming.id); + let mut phase = derive_phase(incoming.status.as_ref()); + let mut sandbox = existing.unwrap_or_else(|| { + use crate::persistence::current_time_ms; + let now_ms = current_time_ms().unwrap_or(0); + Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: incoming.id.clone(), + name: incoming.name.clone(), + created_at_ms: now_ms, + labels: std::collections::HashMap::new(), + }), + spec: None, + status: None, + phase: SandboxPhase::Unknown as i32, + current_policy_version: 0, + } }); + if session_connected && matches!(phase, SandboxPhase::Provisioning | SandboxPhase::Unknown) + { + ensure_supervisor_ready_status(&mut status, sandbox.object_name()); + phase = SandboxPhase::Ready; + } + let old_phase = SandboxPhase::try_from(sandbox.phase).unwrap_or(SandboxPhase::Unknown); if old_phase != phase { info!( @@ -468,8 +864,11 @@ impl ComputeRuntime { } } - sandbox.name = incoming.name; - sandbox.namespace = incoming.namespace; + // Update metadata fields + if let Some(metadata) = sandbox.metadata.as_mut() { + metadata.name = incoming.name; + } + // Note: namespace field removed from public Sandbox API - it remains internal to DriverSandbox sandbox.status = status; sandbox.phase = phase as i32; @@ -482,7 +881,57 @@ impl ComputeRuntime { .put_message(&sandbox) .await .map_err(|e| e.to_string())?; - self.sandbox_watch_bus.notify(&sandbox.id); + self.sandbox_watch_bus.notify(sandbox.object_id()); + Ok(()) + } + + pub async fn supervisor_session_connected(&self, sandbox_id: &str) -> Result<(), String> { + self.set_supervisor_session_state(sandbox_id, true).await + } + + pub async fn supervisor_session_disconnected(&self, sandbox_id: &str) -> Result<(), String> { + self.set_supervisor_session_state(sandbox_id, false).await + } + + async fn set_supervisor_session_state( + &self, + sandbox_id: &str, + connected: bool, + ) -> Result<(), String> { + let _guard = self.sync_lock.lock().await; + let Some(record) = self + .store + .get(Sandbox::object_type(), sandbox_id) + .await + .map_err(|e| e.to_string())? + else { + return Ok(()); + }; + + let mut sandbox = decode_sandbox_record(&record)?; + let current_phase = SandboxPhase::try_from(sandbox.phase).unwrap_or(SandboxPhase::Unknown); + + if current_phase == SandboxPhase::Deleting || current_phase == SandboxPhase::Error { + return Ok(()); + } + + let sandbox_name = sandbox.object_name().to_string(); + if connected { + ensure_supervisor_ready_status(&mut sandbox.status, &sandbox_name); + sandbox.phase = SandboxPhase::Ready as i32; + } else if current_phase == SandboxPhase::Ready { + ensure_supervisor_not_ready_status(&mut sandbox.status, &sandbox_name); + sandbox.phase = SandboxPhase::Provisioning as i32; + } else { + return Ok(()); + } + + self.sandbox_index.update_from_sandbox(&sandbox); + self.store + .put_message(&sandbox) + .await + .map_err(|e| e.to_string())?; + self.sandbox_watch_bus.notify(sandbox_id); Ok(()) } @@ -492,6 +941,15 @@ impl ComputeRuntime { } async fn apply_deleted_locked(&self, sandbox_id: &str) -> Result<(), String> { + let sandbox = self + .store + .get_message::(sandbox_id) + .await + .map_err(|e| e.to_string())?; + if let Some(sandbox) = sandbox.as_ref() { + self.cleanup_sandbox_owned_records(sandbox).await; + } + let _ = self .store .delete(Sandbox::object_type(), sandbox_id) @@ -503,6 +961,44 @@ impl ComputeRuntime { Ok(()) } + async fn cleanup_sandbox_owned_records(&self, sandbox: &Sandbox) { + self.cleanup_sandbox_ssh_sessions(sandbox.object_id()).await; + + if let Err(e) = self + .store + .delete_by_name(SANDBOX_SETTINGS_OBJECT_TYPE, sandbox.object_name()) + .await + { + warn!( + sandbox_id = %sandbox.object_id(), + sandbox_name = %sandbox.object_name(), + error = %e, + "Failed to delete sandbox settings during cleanup" + ); + } + } + + async fn cleanup_sandbox_ssh_sessions(&self, sandbox_id: &str) { + if let Ok(records) = self.store.list(SshSession::object_type(), 1000, 0).await { + for record in records { + if let Ok(session) = SshSession::decode(record.payload.as_slice()) + && session.sandbox_id == sandbox_id + && let Err(e) = self + .store + .delete(SshSession::object_type(), session.object_id()) + .await + { + warn!( + sandbox_id = %sandbox_id, + session_id = %session.object_id(), + error = %e, + "Failed to delete SSH session during sandbox cleanup" + ); + } + } + } + } + fn cleanup_sandbox_state(&self, sandbox_id: &str) { self.tracing_log_bus.remove(sandbox_id); self.tracing_log_bus.platform_event_bus.remove(sandbox_id); @@ -565,19 +1061,22 @@ impl ComputeRuntime { return Ok(()); } - if let Some(current) = self.get_driver_sandbox(&sandbox.id, &sandbox.name).await? { + if let Some(current) = self + .get_driver_sandbox(sandbox.object_id(), sandbox.object_name()) + .await? + { return self .apply_sandbox_update_locked(current, Some(current_record)) .await; } info!( - sandbox_id = %sandbox.id, - sandbox_name = %sandbox.name, + sandbox_id = %sandbox.object_id(), + sandbox_name = %sandbox.object_name(), age_secs = age_ms / 1000, "Removing sandbox from store after it disappeared from the compute driver snapshot" ); - self.apply_deleted_locked(&sandbox.id).await + self.apply_deleted_locked(sandbox.object_id()).await } async fn get_driver_sandbox( @@ -602,9 +1101,9 @@ impl ComputeRuntime { fn driver_sandbox_from_public(sandbox: &Sandbox) -> DriverSandbox { DriverSandbox { - id: sandbox.id.clone(), - name: sandbox.name.clone(), - namespace: sandbox.namespace.clone(), + id: sandbox.object_id().to_string(), + name: sandbox.object_name().to_string(), + namespace: String::new(), // Namespace is set by the driver based on its config spec: sandbox.spec.as_ref().map(driver_sandbox_spec_from_public), status: sandbox .status @@ -622,6 +1121,7 @@ fn driver_sandbox_spec_from_public(spec: &SandboxSpec) -> DriverSandboxSpec { .as_ref() .map(driver_sandbox_template_from_public), gpu: spec.gpu, + gpu_device: spec.gpu_device.clone(), } } @@ -644,8 +1144,6 @@ fn driver_sandbox_template_from_public(template: &SandboxTemplate) -> DriverSand fn extract_typed_resources( resources: &Option, ) -> Option { - let s = resources.as_ref()?; - fn get_quantity(s: &prost_types::Struct, section: &str, key: &str) -> String { s.fields .get(section) @@ -660,6 +1158,8 @@ fn extract_typed_resources( .unwrap_or_default() } + let s = resources.as_ref()?; + let req = DriverResourceRequirements { cpu_request: get_quantity(s, "requests", "cpu"), cpu_limit: get_quantity(s, "limits", "cpu"), @@ -681,7 +1181,7 @@ fn extract_typed_resources( } /// Build the opaque `platform_config` Struct from platform-specific public -/// template fields (runtime_class_name, annotations, volume_claim_templates) +/// template fields (`runtime_class_name`, annotations, `volume_claim_templates`) /// plus any resource fields beyond CPU/memory. fn build_platform_config(template: &SandboxTemplate) -> Option { use prost_types::{Struct, Value, value::Kind}; @@ -730,14 +1230,14 @@ fn build_platform_config(template: &SandboxTemplate) -> Option Option, +) -> Option { + use prost_types::{Struct, Value, value::Kind}; + + let resources = resources.as_ref()?; + let mut fields = std::collections::BTreeMap::new(); + + for (section_name, value) in &resources.fields { + if !matches!(section_name.as_str(), "limits" | "requests") { + fields.insert(section_name.clone(), value.clone()); + continue; + } + + let Some(Kind::StructValue(section)) = value.kind.as_ref() else { + fields.insert(section_name.clone(), value.clone()); + continue; + }; + + let section_fields = section + .fields + .iter() + .filter_map(|(resource_name, resource_value)| { + let is_typed_quantity = matches!(resource_name.as_str(), "cpu" | "memory") + && matches!(resource_value.kind.as_ref(), Some(Kind::StringValue(_))); + if is_typed_quantity { + None + } else { + Some((resource_name.clone(), resource_value.clone())) + } + }) + .collect::>(); + + if !section_fields.is_empty() { + fields.insert( + section_name.clone(), + Value { + kind: Some(Kind::StructValue(Struct { + fields: section_fields, + })), + }, + ); + } + } + + if fields.is_empty() { + None + } else { + Some(Struct { fields }) + } +} + fn driver_status_from_public(status: &SandboxStatus, phase: i32) -> DriverSandboxStatus { DriverSandboxStatus { sandbox_name: status.sandbox_name.clone(), @@ -780,18 +1332,6 @@ impl ObjectType for Sandbox { } } -impl ObjectId for Sandbox { - fn object_id(&self) -> &str { - &self.id - } -} - -impl ObjectName for Sandbox { - fn object_name(&self) -> &str { - &self.name - } -} - fn compute_error_from_status(status: Status) -> ComputeError { match status.code() { Code::AlreadyExists => ComputeError::AlreadyExists, @@ -813,30 +1353,6 @@ fn decode_sandbox_record(record: &ObjectRecord) -> Result { Sandbox::decode(record.payload.as_slice()).map_err(|e| e.to_string()) } -fn resolved_endpoint_from_response( - response: &ResolveSandboxEndpointResponse, -) -> Result { - let endpoint = response - .endpoint - .as_ref() - .ok_or_else(|| ComputeError::Message("compute driver returned no endpoint".to_string()))?; - let port = u16::try_from(endpoint.port) - .map_err(|_| ComputeError::Message("compute driver returned invalid port".to_string()))?; - - match endpoint.target.as_ref() { - Some(sandbox_endpoint::Target::Ip(ip)) => ip - .parse() - .map(|ip| ResolvedEndpoint::Ip(ip, port)) - .map_err(|e| ComputeError::Message(format!("invalid endpoint IP: {e}"))), - Some(sandbox_endpoint::Target::Host(host)) => { - Ok(ResolvedEndpoint::Host(host.clone(), port)) - } - None => Err(ComputeError::Message( - "compute driver returned endpoint without target".to_string(), - )), - } -} - fn public_status_from_driver(status: &DriverSandboxStatus) -> SandboxStatus { SandboxStatus { sandbox_name: status.sandbox_name.clone(), @@ -851,6 +1367,58 @@ fn public_status_from_driver(status: &DriverSandboxStatus) -> SandboxStatus { } } +fn ensure_supervisor_ready_status(status: &mut Option, sandbox_name: &str) { + upsert_ready_condition( + status, + sandbox_name, + SandboxCondition { + r#type: "Ready".to_string(), + status: "True".to_string(), + reason: "DependenciesReady".to_string(), + message: "Supervisor session connected".to_string(), + last_transition_time: String::new(), + }, + ); +} + +fn ensure_supervisor_not_ready_status(status: &mut Option, sandbox_name: &str) { + upsert_ready_condition( + status, + sandbox_name, + SandboxCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: "DependenciesNotReady".to_string(), + message: "Supervisor session disconnected".to_string(), + last_transition_time: String::new(), + }, + ); +} + +fn upsert_ready_condition( + status: &mut Option, + sandbox_name: &str, + condition: SandboxCondition, +) { + let status = status.get_or_insert_with(|| SandboxStatus { + sandbox_name: sandbox_name.to_string(), + agent_pod: String::new(), + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: Vec::new(), + }); + + if let Some(existing) = status + .conditions + .iter_mut() + .find(|existing| existing.r#type == "Ready") + { + *existing = condition; + } else { + status.conditions.push(condition); + } +} + fn public_condition_from_driver(condition: &DriverCondition) -> SandboxCondition { SandboxCondition { r#type: condition.r#type.clone(), @@ -905,22 +1473,153 @@ fn rewrite_user_facing_conditions(status: &mut Option, spec: Opti return; } - if let Some(status) = status { - for condition in &mut status.conditions { - if condition.r#type == "Ready" - && condition.status.eq_ignore_ascii_case("false") - && condition.reason.eq_ignore_ascii_case("Unschedulable") - { - condition.message = "GPU sandbox could not be scheduled on the active gateway. Another GPU sandbox may already be using the available GPU, or the gateway may not currently be able to satisfy GPU placement. Please refer to documentation and use `openshell doctor` commands to inspect GPU support and gateway configuration.".to_string(); - } - } + if let Some(status) = status { + for condition in &mut status.conditions { + if condition.r#type == "Ready" + && condition.status.eq_ignore_ascii_case("false") + && condition.reason.eq_ignore_ascii_case("Unschedulable") + { + condition.message = "GPU sandbox could not be scheduled on the active gateway. Another GPU sandbox may already be using the available GPU, or the gateway may not currently be able to satisfy GPU placement. Please refer to documentation and use `openshell doctor` commands to inspect GPU support and gateway configuration.".to_string(); + } + } + } +} + +/// Phases for which a sandbox should have a running compute resource. +/// `Deleting` and `Error` are intentionally excluded: deletion is in +/// progress, or the sandbox has already failed and should not be +/// silently revived. +fn sandbox_phase_should_be_running(phase: SandboxPhase) -> bool { + matches!( + phase, + SandboxPhase::Provisioning | SandboxPhase::Ready | SandboxPhase::Unknown + ) +} + +fn is_terminal_failure_reason(reason: &str) -> bool { + let reason = reason.to_ascii_lowercase(); + let transient_reasons = [ + "reconcilererror", + "dependenciesnotready", + "starting", + "containerstarting", + "healthcheckstarting", + "inspectfailed", + ]; + !transient_reasons.contains(&reason.as_str()) +} + +#[cfg(test)] +#[derive(Debug, Default)] +pub struct NoopTestDriver; + +#[cfg(test)] +#[tonic::async_trait] +impl ComputeDriver for NoopTestDriver { + type WatchSandboxesStream = DriverWatchStream; + + async fn get_capabilities( + &self, + _request: Request, + ) -> Result, Status> + { + Ok(tonic::Response::new( + openshell_core::proto::compute::v1::GetCapabilitiesResponse { + driver_name: "noop-test-driver".to_string(), + driver_version: "test".to_string(), + default_image: "openshell/sandbox:test".to_string(), + supports_gpu: false, + gpu_count: 0, + }, + )) + } + + async fn validate_sandbox_create( + &self, + _request: Request, + ) -> Result< + tonic::Response, + Status, + > { + Ok(tonic::Response::new( + openshell_core::proto::compute::v1::ValidateSandboxCreateResponse {}, + )) + } + + async fn get_sandbox( + &self, + _request: Request, + ) -> Result, Status> + { + Err(Status::not_found("sandbox not found")) + } + + async fn list_sandboxes( + &self, + _request: Request, + ) -> Result, Status> + { + Ok(tonic::Response::new( + openshell_core::proto::compute::v1::ListSandboxesResponse { + sandboxes: Vec::new(), + }, + )) + } + + async fn create_sandbox( + &self, + _request: Request, + ) -> Result, Status> + { + Ok(tonic::Response::new( + openshell_core::proto::compute::v1::CreateSandboxResponse {}, + )) + } + + async fn stop_sandbox( + &self, + _request: Request, + ) -> Result, Status> + { + Ok(tonic::Response::new( + openshell_core::proto::compute::v1::StopSandboxResponse {}, + )) + } + + async fn delete_sandbox( + &self, + _request: Request, + ) -> Result, Status> + { + Ok(tonic::Response::new( + openshell_core::proto::compute::v1::DeleteSandboxResponse { deleted: true }, + )) + } + + async fn watch_sandboxes( + &self, + _request: Request, + ) -> Result, Status> { + Ok(tonic::Response::new(Box::pin(futures::stream::empty()))) } } -fn is_terminal_failure_reason(reason: &str) -> bool { - let reason = reason.to_ascii_lowercase(); - let transient_reasons = ["reconcilererror", "dependenciesnotready"]; - !transient_reasons.contains(&reason.as_str()) +#[cfg(test)] +pub async fn new_test_runtime(store: Arc) -> ComputeRuntime { + ComputeRuntime { + driver: Arc::new(NoopTestDriver), + shutdown_cleanup: None, + startup_resume: None, + _driver_process: None, + default_image: "openshell/sandbox:test".to_string(), + store, + sandbox_index: SandboxIndex::new(), + sandbox_watch_bus: SandboxWatchBus::new(), + tracing_log_bus: TracingLogBus::new(), + supervisor_sessions: Arc::new(SupervisorSessionRegistry::new()), + sync_lock: Arc::new(Mutex::new(())), + gateway_bind_addresses: Vec::new(), + } } #[cfg(test)] @@ -929,16 +1628,41 @@ mod tests { use futures::stream; use openshell_core::proto::compute::v1::{ CreateSandboxResponse, DeleteSandboxResponse, GetCapabilitiesResponse, GetSandboxRequest, - GetSandboxResponse, ResolveSandboxEndpointResponse, SandboxEndpoint, StopSandboxRequest, - StopSandboxResponse, ValidateSandboxCreateResponse, sandbox_endpoint, + GetSandboxResponse, StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateResponse, }; + use std::collections::HashMap; use std::sync::Arc; + use tokio::sync::{mpsc, oneshot}; + + fn string_value(value: &str) -> prost_types::Value { + prost_types::Value { + kind: Some(prost_types::value::Kind::StringValue(value.to_string())), + } + } + + fn number_value(value: f64) -> prost_types::Value { + prost_types::Value { + kind: Some(prost_types::value::Kind::NumberValue(value)), + } + } + + fn struct_value( + fields: impl IntoIterator, prost_types::Value)>, + ) -> prost_types::Value { + prost_types::Value { + kind: Some(prost_types::value::Kind::StructValue(prost_types::Struct { + fields: fields + .into_iter() + .map(|(key, value)| (key.into(), value)) + .collect(), + })), + } + } #[derive(Debug, Default)] struct TestDriver { listed_sandboxes: Vec, current_sandboxes: Vec, - resolve_precondition: Option, } #[tonic::async_trait] @@ -954,6 +1678,7 @@ mod tests { driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), supports_gpu: true, + gpu_count: 0, })) } @@ -1031,24 +1756,6 @@ mod tests { })) } - async fn resolve_sandbox_endpoint( - &self, - _request: Request, - ) -> Result, Status> { - if let Some(message) = &self.resolve_precondition { - return Err(Status::failed_precondition(message.clone())); - } - - Ok(tonic::Response::new(ResolveSandboxEndpointResponse { - endpoint: Some(SandboxEndpoint { - target: Some(sandbox_endpoint::Target::Host( - "sandbox.default.svc.cluster.local".to_string(), - )), - port: 2222, - }), - })) - } - async fn watch_sandboxes( &self, _request: Request, @@ -1058,28 +1765,69 @@ mod tests { } async fn test_runtime(driver: SharedComputeDriver) -> ComputeRuntime { + test_runtime_with_resume(driver, None).await + } + + async fn test_runtime_with_resume( + driver: SharedComputeDriver, + startup_resume: Option>, + ) -> ComputeRuntime { let store = Arc::new(Store::connect("sqlite::memory:").await.unwrap()); ComputeRuntime { driver, + shutdown_cleanup: None, + startup_resume, + _driver_process: None, default_image: "openshell/sandbox:test".to_string(), store, sandbox_index: SandboxIndex::new(), sandbox_watch_bus: SandboxWatchBus::new(), tracing_log_bus: TracingLogBus::new(), + supervisor_sessions: Arc::new(SupervisorSessionRegistry::new()), sync_lock: Arc::new(Mutex::new(())), + gateway_bind_addresses: Vec::new(), } } + fn register_test_supervisor_session(runtime: &ComputeRuntime, sandbox_id: &str) { + let (tx, _rx) = mpsc::channel(1); + let (shutdown_tx, _shutdown_rx) = oneshot::channel(); + runtime.supervisor_sessions.register( + sandbox_id.to_string(), + "session-1".to_string(), + tx, + shutdown_tx, + ); + } + fn sandbox_record(id: &str, name: &str, phase: SandboxPhase) -> Sandbox { Sandbox { - id: id.to_string(), - name: name.to_string(), - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: id.to_string(), + name: name.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), phase: phase as i32, ..Default::default() } } + fn ssh_session_record(id: &str, sandbox_id: &str) -> SshSession { + SshSession { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: id.to_string(), + name: format!("session-{id}"), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), + sandbox_id: sandbox_id.to_string(), + token: format!("token-{id}"), + revoked: false, + expires_at_ms: 0, + } + } + fn make_driver_condition(reason: &str, message: &str) -> DriverCondition { DriverCondition { r#type: "Ready".to_string(), @@ -1134,6 +1882,7 @@ mod tests { "Pod exists with phase: Pending; Service Exists", ), ("dependenciesnotready", "lowercase also works"), + ("Starting", "VM is starting"), ]; for (reason, message) in transient_cases { @@ -1170,6 +1919,7 @@ mod tests { "DependenciesNotReady", "Pod exists with phase: Pending; Service Exists", ), + ("Starting", "VM is starting"), ]; for (reason, message) in transient_conditions { @@ -1208,6 +1958,123 @@ mod tests { assert_eq!(derive_phase(Some(&status)), SandboxPhase::Ready); } + #[test] + fn build_platform_config_omits_typed_cpu_and_memory_resources() { + let template = SandboxTemplate { + resources: Some(prost_types::Struct { + fields: [ + ( + "limits", + struct_value([("cpu", string_value("2")), ("memory", string_value("1Gi"))]), + ), + ( + "requests", + struct_value([ + ("cpu", string_value("500m")), + ("memory", string_value("512Mi")), + ]), + ), + ] + .into_iter() + .map(|(key, value)| (key.to_string(), value)) + .collect(), + }), + ..Default::default() + }; + + assert!(build_platform_config(&template).is_none()); + } + + #[test] + fn build_platform_config_preserves_non_typed_resource_fields() { + let template = SandboxTemplate { + resources: Some(prost_types::Struct { + fields: [ + ( + "limits", + struct_value([ + ("cpu", string_value("2")), + ("memory", string_value("1Gi")), + ("nvidia.com/gpu", string_value("1")), + ]), + ), + ( + "requests", + struct_value([ + ("cpu", string_value("500m")), + ("memory", string_value("512Mi")), + ("hugepages-2Mi", string_value("4Mi")), + ]), + ), + ("opaque_cpu", number_value(2.0)), + ] + .into_iter() + .map(|(key, value)| (key.to_string(), value)) + .collect(), + }), + ..Default::default() + }; + + let platform_config = build_platform_config(&template).unwrap(); + let resources_raw = platform_config + .fields + .get("resources_raw") + .and_then(|value| value.kind.as_ref()) + .and_then(|kind| match kind { + prost_types::value::Kind::StructValue(inner) => Some(inner), + _ => None, + }) + .unwrap(); + + let limits = resources_raw + .fields + .get("limits") + .and_then(|value| value.kind.as_ref()) + .and_then(|kind| match kind { + prost_types::value::Kind::StructValue(inner) => Some(inner), + _ => None, + }) + .unwrap(); + assert!(!limits.fields.contains_key("cpu")); + assert!(!limits.fields.contains_key("memory")); + assert_eq!( + limits + .fields + .get("nvidia.com/gpu") + .and_then(|value| value.kind.as_ref()) + .and_then(|kind| match kind { + prost_types::value::Kind::StringValue(value) => Some(value.as_str()), + _ => None, + }), + Some("1") + ); + + let requests = resources_raw + .fields + .get("requests") + .and_then(|value| value.kind.as_ref()) + .and_then(|kind| match kind { + prost_types::value::Kind::StructValue(inner) => Some(inner), + _ => None, + }) + .unwrap(); + assert!(!requests.fields.contains_key("cpu")); + assert!(!requests.fields.contains_key("memory")); + assert_eq!( + requests + .fields + .get("hugepages-2Mi") + .and_then(|value| value.kind.as_ref()) + .and_then(|kind| match kind { + prost_types::value::Kind::StringValue(value) => Some(value.as_str()), + _ => None, + }), + Some("4Mi") + ); + + assert!(resources_raw.fields.contains_key("opaque_cpu")); + } + #[test] fn rewrite_user_facing_conditions_rewrites_gpu_unschedulable_message() { let mut status = Some(SandboxStatus { @@ -1323,20 +2190,119 @@ mod tests { } #[tokio::test] - async fn resolve_sandbox_endpoint_preserves_precondition_errors() { - let runtime = test_runtime(Arc::new(TestDriver { - resolve_precondition: Some("sandbox agent pod IP is not available".to_string()), - ..Default::default() - })) - .await; + async fn apply_sandbox_update_promotes_connected_supervisor_session_to_ready() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Provisioning); + runtime.store.put_message(&sandbox).await.unwrap(); + + register_test_supervisor_session(&runtime, "sb-1"); + + runtime + .apply_sandbox_update(DriverSandbox { + id: "sb-1".to_string(), + name: "sandbox-a".to_string(), + namespace: "default".to_string(), + spec: None, + status: Some(make_driver_status(make_driver_condition( + "Starting", + "VM is starting", + ))), + }) + .await + .unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase).unwrap(), + SandboxPhase::Ready + ); + let ready = stored + .status + .as_ref() + .and_then(|status| { + status + .conditions + .iter() + .find(|condition| condition.r#type == "Ready") + }) + .unwrap(); + assert_eq!(ready.status, "True"); + assert_eq!(ready.reason, "DependenciesReady"); + assert_eq!(ready.message, "Supervisor session connected"); + } + + #[tokio::test] + async fn supervisor_session_connected_promotes_store_state_without_driver_refresh() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Provisioning); + runtime.store.put_message(&sandbox).await.unwrap(); + + runtime.supervisor_session_connected("sb-1").await.unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase).unwrap(), + SandboxPhase::Ready + ); + } + + #[tokio::test] + async fn supervisor_session_disconnected_demotes_ready_sandbox() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let mut sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + sandbox.status = Some(SandboxStatus { + sandbox_name: "sandbox-a".to_string(), + agent_pod: String::new(), + agent_fd: String::new(), + sandbox_fd: String::new(), + conditions: vec![SandboxCondition { + r#type: "Ready".to_string(), + status: "True".to_string(), + reason: "DependenciesReady".to_string(), + message: "Supervisor session connected".to_string(), + last_transition_time: String::new(), + }], + }); + runtime.store.put_message(&sandbox).await.unwrap(); - let err = runtime - .resolve_sandbox_endpoint(&sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready)) + runtime + .supervisor_session_disconnected("sb-1") .await - .expect_err("endpoint resolution should preserve failed-precondition errors"); + .unwrap(); - assert_eq!(err.code(), Code::FailedPrecondition); - assert_eq!(err.message(), "sandbox agent pod IP is not available"); + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase).unwrap(), + SandboxPhase::Provisioning + ); + let ready = stored + .status + .as_ref() + .and_then(|status| { + status + .conditions + .iter() + .find(|condition| condition.r#type == "Ready") + }) + .unwrap(); + assert_eq!(ready.status, "False"); + assert_eq!(ready.reason, "DependenciesNotReady"); + assert_eq!(ready.message, "Supervisor session disconnected"); } #[tokio::test] @@ -1382,7 +2348,6 @@ mod tests { deleting: false, }), }], - ..Default::default() })) .await; @@ -1440,7 +2405,6 @@ mod tests { last_transition_time: String::new(), })), }], - ..Default::default() })) .await; @@ -1513,8 +2477,21 @@ mod tests { let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Provisioning); runtime.store.put_message(&sandbox).await.unwrap(); runtime.sandbox_index.update_from_sandbox(&sandbox); + runtime + .store + .put( + SANDBOX_SETTINGS_OBJECT_TYPE, + "settings-sb-1", + sandbox.object_name(), + br#"{"revision":1,"settings":{}}"#, + None, + ) + .await + .unwrap(); + let session = ssh_session_record("session-1", sandbox.object_id()); + runtime.store.put_message(&session).await.unwrap(); - let mut watch_rx = runtime.sandbox_watch_bus.subscribe(&sandbox.id); + let mut watch_rx = runtime.sandbox_watch_bus.subscribe(sandbox.object_id()); runtime .reconcile_store_with_backend(Duration::ZERO) @@ -1524,7 +2501,7 @@ mod tests { assert!( runtime .store - .get_message::(&sandbox.id) + .get_message::(sandbox.object_id()) .await .unwrap() .is_none() @@ -1532,7 +2509,23 @@ mod tests { assert!( runtime .sandbox_index - .sandbox_id_for_sandbox_name(&sandbox.name) + .sandbox_id_for_sandbox_name(sandbox.object_name()) + .is_none() + ); + assert!( + runtime + .store + .get_by_name(SANDBOX_SETTINGS_OBJECT_TYPE, sandbox.object_name()) + .await + .unwrap() + .is_none() + ); + assert!( + runtime + .store + .get_message::(session.object_id()) + .await + .unwrap() .is_none() ); let _ = watch_rx.try_recv(); @@ -1541,4 +2534,162 @@ mod tests { Err(tokio::sync::broadcast::error::TryRecvError::Closed) )); } + + #[derive(Default)] + struct RecordingResume { + calls: Mutex>, + results: Mutex>>, + } + + impl RecordingResume { + async fn set_result(&self, sandbox_id: &str, result: Result) { + self.results + .lock() + .await + .insert(sandbox_id.to_string(), result); + } + + async fn calls(&self) -> Vec<(String, String)> { + self.calls.lock().await.clone() + } + } + + #[tonic::async_trait] + impl StartupResume for RecordingResume { + async fn resume_sandbox( + &self, + sandbox_id: &str, + sandbox_name: &str, + ) -> Result { + self.calls + .lock() + .await + .push((sandbox_id.to_string(), sandbox_name.to_string())); + self.results + .lock() + .await + .get(sandbox_id) + .cloned() + .unwrap_or(Ok(true)) + } + } + + #[tokio::test] + async fn resume_persisted_sandboxes_resumes_running_phases() { + let resume = Arc::new(RecordingResume::default()); + let runtime = + test_runtime_with_resume(Arc::new(TestDriver::default()), Some(resume.clone())).await; + + for (id, name, phase) in [ + ("sb-prov", "prov", SandboxPhase::Provisioning), + ("sb-ready", "ready", SandboxPhase::Ready), + ("sb-unknown", "unknown", SandboxPhase::Unknown), + ("sb-deleting", "deleting", SandboxPhase::Deleting), + ("sb-error", "error", SandboxPhase::Error), + ] { + let sandbox = sandbox_record(id, name, phase); + runtime.store.put_message(&sandbox).await.unwrap(); + } + + runtime.resume_persisted_sandboxes().await.unwrap(); + + let mut called_ids = resume + .calls() + .await + .into_iter() + .map(|(id, _)| id) + .collect::>(); + called_ids.sort(); + assert_eq!( + called_ids, + vec![ + "sb-prov".to_string(), + "sb-ready".to_string(), + "sb-unknown".to_string(), + ] + ); + } + + #[tokio::test] + async fn resume_persisted_sandboxes_marks_missing_backend_as_error() { + let resume = Arc::new(RecordingResume::default()); + resume.set_result("sb-1", Ok(false)).await; + let runtime = + test_runtime_with_resume(Arc::new(TestDriver::default()), Some(resume.clone())).await; + + let sandbox = sandbox_record("sb-1", "missing", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + runtime.resume_persisted_sandboxes().await.unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase).unwrap(), + SandboxPhase::Error + ); + let ready = stored + .status + .as_ref() + .and_then(|s| s.conditions.iter().find(|c| c.r#type == "Ready")) + .expect("Ready condition present"); + assert_eq!(ready.reason, "BackendResourceMissing"); + } + + #[tokio::test] + async fn resume_persisted_sandboxes_marks_failed_resume_as_error() { + let resume = Arc::new(RecordingResume::default()); + resume + .set_result("sb-1", Err("docker daemon angry".to_string())) + .await; + let runtime = + test_runtime_with_resume(Arc::new(TestDriver::default()), Some(resume.clone())).await; + + let sandbox = sandbox_record("sb-1", "broken", SandboxPhase::Provisioning); + runtime.store.put_message(&sandbox).await.unwrap(); + + runtime.resume_persisted_sandboxes().await.unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase).unwrap(), + SandboxPhase::Error + ); + let ready = stored + .status + .as_ref() + .and_then(|s| s.conditions.iter().find(|c| c.r#type == "Ready")) + .expect("Ready condition present"); + assert_eq!(ready.reason, "ResumeFailed"); + assert!(ready.message.contains("docker daemon angry")); + } + + #[tokio::test] + async fn resume_persisted_sandboxes_is_noop_without_resume_hook() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-1", "anywhere", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + runtime.resume_persisted_sandboxes().await.unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase).unwrap(), + SandboxPhase::Ready + ); + } } diff --git a/crates/openshell-server/src/compute/vm.rs b/crates/openshell-server/src/compute/vm.rs new file mode 100644 index 0000000000..e5b974f745 --- /dev/null +++ b/crates/openshell-server/src/compute/vm.rs @@ -0,0 +1,537 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! VM compute driver plumbing. +//! +//! This module owns everything needed to hand the gateway a `Channel` speaking +//! the `openshell.compute.v1.ComputeDriver` RPC surface against an +//! `openshell-driver-vm` subprocess over a Unix domain socket: +//! +//! - [`VmComputeConfig`]: gateway-local configuration (state dir, driver binary, +//! VM shape, guest TLS material). +//! - [`spawn`]: spawn the driver subprocess, wait for its UDS to be ready, +//! and return a live gRPC channel plus a [`ManagedDriverProcess`] handle +//! that will reap the subprocess and clean up the socket on drop. +//! - Helpers to resolve the driver binary, compute the socket path, and +//! validate guest TLS material when the gateway runs an `https://` control +//! plane. +//! +//! The VM-driver fields deliberately live here rather than in +//! [`openshell_core::Config`] so the shared core stays free of driver-specific +//! plumbing. +//! +//! TODO(driver-abstraction): this module still assumes the concrete VM driver +//! (argv shape, guest-TLS flags, libkrun-specific settings). Once we land the +//! generalized compute-driver interface, the CLI-arg plumbing below should +//! be replaced with a driver-agnostic launcher that speaks gRPC to +//! configure the driver — and this file should collapse to the types that +//! are genuinely VM-specific (libkrun log level, vCPU / memory shape) plus a +//! trait implementation registering the VM driver against the generic +//! interface. + +#[cfg(unix)] +use super::ManagedDriverProcess; +#[cfg(unix)] +use hyper_util::rt::TokioIo; +#[cfg(unix)] +use openshell_core::proto::compute::v1::{ + GetCapabilitiesRequest, compute_driver_client::ComputeDriverClient, +}; +use openshell_core::{Config, Error, Result}; +use std::path::PathBuf; +#[cfg(unix)] +use std::{io::ErrorKind, process::Stdio, sync::Arc, time::Duration}; +#[cfg(unix)] +use tokio::net::UnixStream; +#[cfg(unix)] +use tokio::process::Command; +use tonic::transport::Channel; +#[cfg(unix)] +use tonic::transport::Endpoint; +#[cfg(unix)] +use tower::service_fn; + +const DRIVER_BIN_NAME: &str = "openshell-driver-vm"; + +/// Configuration for launching and talking to the VM compute driver. +#[derive(Debug, Clone)] +pub struct VmComputeConfig { + /// Working directory for VM driver sandbox state. + pub state_dir: PathBuf, + + /// Directory to search for compute-driver binaries before the gateway + /// falls back to its conventional install paths and sibling binary. + pub driver_dir: Option, + + /// Default sandbox image the driver should use when a request omits one. + pub default_image: String, + + /// libkrun log level used by the VM driver helper. + pub krun_log_level: u32, + + /// Default vCPU count for VM sandboxes. + pub vcpus: u8, + + /// Default memory allocation for VM sandboxes, in MiB. + pub mem_mib: u32, + + /// Host-side CA certificate for the guest's mTLS client bundle. + pub guest_tls_ca: Option, + + /// Host-side client certificate for the guest's mTLS client bundle. + pub guest_tls_cert: Option, + + /// Host-side private key for the guest's mTLS client bundle. + pub guest_tls_key: Option, +} + +impl VmComputeConfig { + /// Default working directory for VM driver state. + #[must_use] + pub fn default_state_dir() -> PathBuf { + PathBuf::from("target/openshell-vm-driver") + } + + /// Default libkrun log level. + #[must_use] + pub const fn default_krun_log_level() -> u32 { + 1 + } + + /// Default vCPU count. + #[must_use] + pub const fn default_vcpus() -> u8 { + 2 + } + + /// Default memory allocation, in MiB. + #[must_use] + pub const fn default_mem_mib() -> u32 { + 2048 + } + + #[must_use] + fn default_driver_search_dirs(home: Option) -> Vec { + let mut dirs = Vec::new(); + if let Some(home) = home { + dirs.push(home.join(".local").join("libexec").join("openshell")); + } + push_unique_path(&mut dirs, PathBuf::from("/usr/libexec/openshell")); + push_unique_path(&mut dirs, PathBuf::from("/usr/local/libexec/openshell")); + push_unique_path(&mut dirs, PathBuf::from("/usr/local/libexec")); + dirs + } +} + +impl Default for VmComputeConfig { + fn default() -> Self { + Self { + state_dir: Self::default_state_dir(), + driver_dir: None, + default_image: String::new(), + krun_log_level: Self::default_krun_log_level(), + vcpus: Self::default_vcpus(), + mem_mib: Self::default_mem_mib(), + guest_tls_ca: None, + guest_tls_cert: None, + guest_tls_key: None, + } + } +} + +#[cfg(unix)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct VmGuestTlsPaths { + pub ca: PathBuf, + pub cert: PathBuf, + pub key: PathBuf, +} + +/// Resolve the `openshell-driver-vm` binary path. +/// +/// Resolution order: +/// 1. `{driver_dir}/openshell-driver-vm`, where `driver_dir` comes from +/// `--driver-dir` / `OPENSHELL_DRIVER_DIR`. +/// 2. Conventional install directories: +/// `~/.local/libexec/openshell`, `/usr/libexec/openshell`, +/// `/usr/local/libexec/openshell`, `/usr/local/libexec`. +/// 3. Sibling of the gateway's own executable (last-resort fallback so +/// local development builds still work out of the box). +pub fn resolve_compute_driver_bin(vm_config: &VmComputeConfig) -> Result { + let mut searched: Vec = Vec::new(); + + // 1. Configured driver directory, or the conventional install locations + // when no explicit override is configured. + for dir in resolve_driver_search_dirs(vm_config) { + let candidate = dir.join(DRIVER_BIN_NAME); + if candidate.is_file() { + return Ok(candidate); + } + push_unique_path(&mut searched, candidate); + } + + // 2. Sibling-of-gateway fallback. + let current_exe = std::env::current_exe() + .map_err(|e| Error::config(format!("failed to resolve current executable: {e}")))?; + let Some(parent) = current_exe.parent() else { + return Err(Error::config(format!( + "current executable '{}' has no parent directory", + current_exe.display() + ))); + }; + let sibling = parent.join(DRIVER_BIN_NAME); + if sibling.is_file() { + return Ok(sibling); + } + push_unique_path(&mut searched, sibling); + + let searched_display = searched + .iter() + .map(|p| format!("'{}'", p.display())) + .collect::>() + .join(", "); + Err(Error::config(format!( + "vm compute driver binary not found (searched {searched_display}); install it under --driver-dir / OPENSHELL_DRIVER_DIR, a conventional libexec path such as ~/.local/libexec/openshell, /usr/libexec/openshell, or /usr/local/libexec{{,/openshell}}, or place it next to the gateway binary" + ))) +} + +fn resolve_driver_search_dirs(vm_config: &VmComputeConfig) -> Vec { + vm_config.driver_dir.clone().map_or_else( + || VmComputeConfig::default_driver_search_dirs(std::env::var_os("HOME").map(PathBuf::from)), + |dir| vec![dir], + ) +} + +fn push_unique_path(paths: &mut Vec, path: PathBuf) { + if !paths.iter().any(|existing| existing == &path) { + paths.push(path); + } +} + +/// Path of the Unix domain socket the driver will listen on. +pub fn compute_driver_socket_path(vm_config: &VmComputeConfig) -> PathBuf { + vm_config.state_dir.join("compute-driver.sock") +} + +#[cfg(unix)] +pub fn compute_driver_guest_tls_paths( + config: &Config, + vm_config: &VmComputeConfig, +) -> Result> { + if !config.grpc_endpoint.starts_with("https://") { + return Ok(None); + } + + let provided = [ + vm_config.guest_tls_ca.as_ref(), + vm_config.guest_tls_cert.as_ref(), + vm_config.guest_tls_key.as_ref(), + ]; + if provided.iter().all(Option::is_none) { + return Err(Error::config( + "vm compute driver requires --vm-tls-ca, --vm-tls-cert, and --vm-tls-key when OPENSHELL_GRPC_ENDPOINT uses https://", + )); + } + + let Some(ca) = vm_config.guest_tls_ca.clone() else { + return Err(Error::config( + "--vm-tls-ca is required when VM guest TLS materials are configured", + )); + }; + let Some(cert) = vm_config.guest_tls_cert.clone() else { + return Err(Error::config( + "--vm-tls-cert is required when VM guest TLS materials are configured", + )); + }; + let Some(key) = vm_config.guest_tls_key.clone() else { + return Err(Error::config( + "--vm-tls-key is required when VM guest TLS materials are configured", + )); + }; + + for path in [&ca, &cert, &key] { + if !path.is_file() { + return Err(Error::config(format!( + "vm guest TLS material '{}' does not exist or is not a file", + path.display() + ))); + } + } + + Ok(Some(VmGuestTlsPaths { ca, cert, key })) +} + +/// Launch the VM compute-driver subprocess, wait for its UDS to come up, +/// and return a gRPC `Channel` connected to it plus a process handle that +/// kills the subprocess and removes the socket on drop. +#[cfg(unix)] +pub async fn spawn( + config: &Config, + vm_config: &VmComputeConfig, +) -> Result<(Channel, Arc)> { + if config.grpc_endpoint.trim().is_empty() { + return Err(Error::config( + "grpc_endpoint is required when using the vm compute driver", + )); + } + + let driver_bin = resolve_compute_driver_bin(vm_config)?; + let socket_path = compute_driver_socket_path(vm_config); + let guest_tls_paths = compute_driver_guest_tls_paths(config, vm_config)?; + if let Some(parent) = socket_path.parent() { + std::fs::create_dir_all(parent).map_err(|e| { + Error::execution(format!( + "failed to create vm compute driver socket dir '{}': {e}", + parent.display() + )) + })?; + } + match std::fs::remove_file(&socket_path) { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => { + return Err(Error::execution(format!( + "failed to remove stale vm compute driver socket '{}': {err}", + socket_path.display() + ))); + } + } + + let mut command = Command::new(&driver_bin); + command.kill_on_drop(true); + command.stdin(Stdio::null()); + command.stdout(Stdio::inherit()); + command.stderr(Stdio::inherit()); + command.arg("--bind-socket").arg(&socket_path); + command.arg("--log-level").arg(&config.log_level); + command + .arg("--openshell-endpoint") + .arg(&config.grpc_endpoint); + command.arg("--state-dir").arg(&vm_config.state_dir); + if !vm_config.default_image.trim().is_empty() { + command.arg("--default-image").arg(&vm_config.default_image); + } + // Only forward the handshake secret when one is configured. The VM + // driver does not consume it, but accepts it for parity with the + // Kubernetes/Podman drivers; passing an empty value is noise. + if !config.ssh_handshake_secret.is_empty() { + command + .arg("--ssh-handshake-secret") + .arg(&config.ssh_handshake_secret); + } + command + .arg("--ssh-handshake-skew-secs") + .arg(config.ssh_handshake_skew_secs.to_string()); + command + .arg("--krun-log-level") + .arg(vm_config.krun_log_level.to_string()); + command.arg("--vcpus").arg(vm_config.vcpus.to_string()); + command.arg("--mem-mib").arg(vm_config.mem_mib.to_string()); + if let Some(tls) = guest_tls_paths { + command.arg("--guest-tls-ca").arg(tls.ca); + command.arg("--guest-tls-cert").arg(tls.cert); + command.arg("--guest-tls-key").arg(tls.key); + } + + let mut child = command.spawn().map_err(|e| { + Error::execution(format!( + "failed to launch vm compute driver '{}': {e}", + driver_bin.display() + )) + })?; + let channel = wait_for_compute_driver(&socket_path, &mut child).await?; + let process = Arc::new(ManagedDriverProcess::new(child, socket_path)); + Ok((channel, process)) +} + +#[cfg(not(unix))] +pub async fn spawn( + _config: &Config, + _vm_config: &VmComputeConfig, +) -> Result<(Channel, std::sync::Arc)> { + Err(Error::config( + "the vm compute driver requires unix domain socket support", + )) +} + +#[cfg(unix)] +async fn wait_for_compute_driver( + socket_path: &std::path::Path, + child: &mut tokio::process::Child, +) -> Result { + let mut last_error: Option = None; + for _ in 0..100 { + let try_wait_result = child.try_wait().map_err(|e| { + Error::execution(format!("failed to poll vm compute driver process: {e}")) + })?; + if let Some(status) = try_wait_result { + return Err(Error::execution(format!( + "vm compute driver exited before becoming ready with status {status}" + ))); + } + + match connect_compute_driver(socket_path).await { + Ok(channel) => { + let mut client = ComputeDriverClient::new(channel.clone()); + match client + .get_capabilities(tonic::Request::new(GetCapabilitiesRequest {})) + .await + { + Ok(_) => return Ok(channel), + Err(status) => last_error = Some(status.to_string()), + } + } + Err(err) => last_error = Some(err.to_string()), + } + + tokio::time::sleep(Duration::from_millis(100)).await; + } + + Err(Error::execution(format!( + "timed out waiting for vm compute driver socket '{}': {}", + socket_path.display(), + last_error.unwrap_or_else(|| "unknown error".to_string()) + ))) +} + +#[cfg(unix)] +async fn connect_compute_driver(socket_path: &std::path::Path) -> Result { + let socket_path = socket_path.to_path_buf(); + let display_path = socket_path.clone(); + Endpoint::from_static("http://[::]:50051") + .connect_with_connector(service_fn(move |_: tonic::transport::Uri| { + let socket_path = socket_path.clone(); + async move { UnixStream::connect(socket_path).await.map(TokioIo::new) } + })) + .await + .map_err(|e| { + Error::execution(format!( + "failed to connect to vm compute driver socket '{}': {e}", + display_path.display() + )) + }) +} + +#[cfg(all(test, unix))] +mod tests { + use super::{ + VmComputeConfig, compute_driver_guest_tls_paths, resolve_compute_driver_bin, + resolve_driver_search_dirs, + }; + use openshell_core::{Config, TlsConfig}; + use std::os::unix::fs::PermissionsExt; + use std::path::PathBuf; + use tempfile::tempdir; + + #[test] + fn resolve_driver_bin_uses_driver_dir_when_binary_present() { + let dir = tempdir().unwrap(); + let bin = dir.path().join("openshell-driver-vm"); + std::fs::write(&bin, "#!/bin/sh\n").unwrap(); + std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).unwrap(); + + let vm_config = VmComputeConfig { + driver_dir: Some(dir.path().to_path_buf()), + ..Default::default() + }; + assert_eq!(resolve_compute_driver_bin(&vm_config).unwrap(), bin); + } + + #[test] + fn resolve_driver_bin_error_mentions_driver_dir_hint() { + let dir = tempdir().unwrap(); // empty — no driver binary present + + let vm_config = VmComputeConfig { + driver_dir: Some(dir.path().to_path_buf()), + ..Default::default() + }; + let err = resolve_compute_driver_bin(&vm_config) + .unwrap_err() + .to_string(); + assert!(err.contains("--driver-dir")); + assert!(err.contains("OPENSHELL_DRIVER_DIR")); + assert!(err.contains("openshell-driver-vm")); + } + + #[test] + fn resolve_driver_search_dirs_include_libexec_fallbacks() { + let dirs = resolve_driver_search_dirs(&VmComputeConfig { + driver_dir: None, + ..Default::default() + }); + + assert!(dirs.contains(&PathBuf::from("/usr/libexec/openshell"))); + assert!(dirs.contains(&PathBuf::from("/usr/local/libexec/openshell"))); + assert!(dirs.contains(&PathBuf::from("/usr/local/libexec"))); + } + + #[test] + fn vm_compute_driver_tls_requires_explicit_guest_bundle() { + let dir = tempdir().unwrap(); + let server_cert = dir.path().join("server.crt"); + let server_key = dir.path().join("server.key"); + let server_ca = dir.path().join("client-ca.crt"); + std::fs::write(&server_cert, "server-cert").unwrap(); + std::fs::write(&server_key, "server-key").unwrap(); + std::fs::write(&server_ca, "client-ca").unwrap(); + + let config = Config::new(Some(TlsConfig { + cert_path: server_cert, + key_path: server_key, + client_ca_path: server_ca, + allow_unauthenticated: false, + })) + .with_grpc_endpoint("https://gateway.internal:8443"); + + let err = compute_driver_guest_tls_paths(&config, &VmComputeConfig::default()) + .expect_err("https vm endpoints should require an explicit guest client bundle"); + assert!( + err.to_string() + .contains("--vm-tls-ca, --vm-tls-cert, and --vm-tls-key") + ); + } + + #[test] + fn vm_compute_driver_tls_uses_guest_bundle_not_gateway_server_identity() { + let dir = tempdir().unwrap(); + let server_cert = dir.path().join("server.crt"); + let server_key = dir.path().join("server.key"); + let server_ca = dir.path().join("client-ca.crt"); + let guest_ca = dir.path().join("guest-ca.crt"); + let guest_cert = dir.path().join("guest.crt"); + let guest_key = dir.path().join("guest.key"); + for path in [ + &server_cert, + &server_key, + &server_ca, + &guest_ca, + &guest_cert, + &guest_key, + ] { + std::fs::write(path, path.display().to_string()).unwrap(); + } + + let config = Config::new(Some(TlsConfig { + cert_path: server_cert.clone(), + key_path: server_key.clone(), + client_ca_path: server_ca, + allow_unauthenticated: false, + })) + .with_grpc_endpoint("https://gateway.internal:8443"); + let vm_config = VmComputeConfig { + guest_tls_ca: Some(guest_ca.clone()), + guest_tls_cert: Some(guest_cert.clone()), + guest_tls_key: Some(guest_key.clone()), + ..Default::default() + }; + + let guest_paths = compute_driver_guest_tls_paths(&config, &vm_config) + .unwrap() + .expect("https vm endpoints should pass an explicit guest client bundle"); + assert_eq!(guest_paths.ca, guest_ca); + assert_eq!(guest_paths.cert, guest_cert); + assert_eq!(guest_paths.key, guest_key); + assert_ne!(guest_paths.cert, server_cert); + assert_ne!(guest_paths.key, server_key); + } +} diff --git a/crates/openshell-server/src/grpc/mod.rs b/crates/openshell-server/src/grpc/mod.rs index af60897d1f..31970e9c5b 100644 --- a/crates/openshell-server/src/grpc/mod.rs +++ b/crates/openshell-server/src/grpc/mod.rs @@ -3,7 +3,7 @@ //! gRPC service implementation. -pub(crate) mod policy; +pub mod policy; mod provider; mod sandbox; mod validation; @@ -14,22 +14,25 @@ use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, EditDraftChunkRequest, EditDraftChunkResponse, ExecSandboxEvent, ExecSandboxRequest, - GetDraftHistoryRequest, GetDraftHistoryResponse, GetDraftPolicyRequest, GetDraftPolicyResponse, - GetGatewayConfigRequest, GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GatewayMessage, GetDraftHistoryRequest, GetDraftHistoryResponse, GetDraftPolicyRequest, + GetDraftPolicyResponse, GetGatewayConfigRequest, GetGatewayConfigResponse, + GetProviderProfileRequest, GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, GetSandboxLogsRequest, GetSandboxLogsResponse, GetSandboxPolicyStatusRequest, GetSandboxPolicyStatusResponse, GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxPoliciesRequest, ListSandboxPoliciesResponse, ListSandboxesRequest, - ListSandboxesResponse, ProviderResponse, PushSandboxLogsRequest, PushSandboxLogsResponse, - RejectDraftChunkRequest, RejectDraftChunkResponse, ReportPolicyStatusRequest, + HealthRequest, HealthResponse, ListProviderProfilesRequest, ListProviderProfilesResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxPoliciesRequest, + ListSandboxPoliciesResponse, ListSandboxesRequest, ListSandboxesResponse, + ProviderProfileResponse, ProviderResponse, PushSandboxLogsRequest, PushSandboxLogsResponse, + RejectDraftChunkRequest, RejectDraftChunkResponse, RelayFrame, ReportPolicyStatusRequest, ReportPolicyStatusResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, ServiceStatus, SubmitPolicyAnalysisRequest, SubmitPolicyAnalysisResponse, - UndoDraftChunkRequest, UndoDraftChunkResponse, UpdateConfigRequest, UpdateConfigResponse, - UpdateProviderRequest, WatchSandboxRequest, open_shell_server::OpenShell, + SupervisorMessage, UndoDraftChunkRequest, UndoDraftChunkResponse, UpdateConfigRequest, + UpdateConfigResponse, UpdateProviderRequest, WatchSandboxRequest, open_shell_server::OpenShell, }; use serde::{Deserialize, Serialize}; use std::collections::BTreeMap; +use std::pin::Pin; use std::sync::Arc; use tokio_stream::wrappers::ReceiverStream; use tonic::{Request, Response, Status}; @@ -114,6 +117,19 @@ fn current_time_ms() -> Result { Ok(i64::try_from(now.as_millis()).unwrap_or(i64::MAX)) } +/// Validate that object metadata is present and contains required fields. +/// +/// This is a crate-level helper that wraps the validation module's implementation. +/// Use this from modules outside of `grpc` that need to validate metadata. +// `tonic::Status` is large but is the API surface of gRPC handlers. +#[allow(clippy::result_large_err)] +pub fn validate_object_metadata( + metadata: Option<&openshell_core::proto::datamodel::v1::ObjectMeta>, + resource_type: &str, +) -> Result<(), Status> { + validation::validate_object_metadata(metadata, resource_type) +} + // --------------------------------------------------------------------------- // Service struct // --------------------------------------------------------------------------- @@ -238,6 +254,23 @@ impl OpenShell for OpenShellService { provider::handle_list_providers(&self.state, request).await } + async fn list_provider_profiles( + &self, + request: Request, + ) -> Result, Status> { + Ok(provider::handle_list_provider_profiles( + &self.state, + request, + )) + } + + async fn get_provider_profile( + &self, + request: Request, + ) -> Result, Status> { + provider::handle_get_provider_profile(&self.state, request) + } + async fn update_provider( &self, request: Request, @@ -383,6 +416,29 @@ impl OpenShell for OpenShellService { ) -> Result, Status> { policy::handle_get_draft_history(&self.state, request).await } + + // --- Supervisor session --- + + type ConnectSupervisorStream = + Pin> + Send + 'static>>; + + async fn connect_supervisor( + &self, + request: Request>, + ) -> Result, Status> { + crate::supervisor_session::handle_connect_supervisor(&self.state, request).await + } + + type RelayStreamStream = + Pin> + Send + 'static>>; + + async fn relay_stream( + &self, + request: Request>, + ) -> Result, Status> { + crate::supervisor_session::handle_relay_stream(&self.state.supervisor_sessions, request) + .await + } } // --------------------------------------------------------------------------- diff --git a/crates/openshell-server/src/grpc/policy.rs b/crates/openshell-server/src/grpc/policy.rs index 58d0c03cf6..6caa81031f 100644 --- a/crates/openshell-server/src/grpc/policy.rs +++ b/crates/openshell-server/src/grpc/policy.rs @@ -10,10 +10,13 @@ #![allow(clippy::cast_precision_loss)] // f64->f32 for confidence scores #![allow(clippy::items_after_statements)] // DB_PORTS const inside function -use crate::ServerState; -use crate::persistence::{DraftChunkRecord, PolicyRecord, Store}; +use crate::persistence::{DraftChunkRecord, ObjectId, ObjectName, PolicyRecord, Store}; +use crate::policy_store::PolicyStoreExt; +use crate::{ServerState, auth::oidc}; +use openshell_core::proto::policy_merge_operation; use openshell_core::proto::setting_value; use openshell_core::proto::{ + AddAllowRules as ProtoAddAllowRules, AddDenyRules as ProtoAddDenyRules, ApproveAllDraftChunksRequest, ApproveAllDraftChunksResponse, ApproveDraftChunkRequest, ApproveDraftChunkResponse, ClearDraftChunksRequest, ClearDraftChunksResponse, DraftHistoryEntry, EditDraftChunkRequest, EditDraftChunkResponse, EffectiveSetting, @@ -22,18 +25,32 @@ use openshell_core::proto::{ GetSandboxConfigResponse, GetSandboxLogsRequest, GetSandboxLogsResponse, GetSandboxPolicyStatusRequest, GetSandboxPolicyStatusResponse, GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, - ListSandboxPoliciesRequest, ListSandboxPoliciesResponse, PolicyChunk, PolicySource, - PolicyStatus, PushSandboxLogsRequest, PushSandboxLogsResponse, RejectDraftChunkRequest, - RejectDraftChunkResponse, ReportPolicyStatusRequest, ReportPolicyStatusResponse, - SandboxLogLine, SandboxPolicyRevision, SettingScope, SettingValue, SubmitPolicyAnalysisRequest, - SubmitPolicyAnalysisResponse, UndoDraftChunkRequest, UndoDraftChunkResponse, - UpdateConfigRequest, UpdateConfigResponse, + ListSandboxPoliciesRequest, ListSandboxPoliciesResponse, PolicyChunk, PolicyMergeOperation, + PolicySource, PolicyStatus, PushSandboxLogsRequest, PushSandboxLogsResponse, + RejectDraftChunkRequest, RejectDraftChunkResponse, ReportPolicyStatusRequest, + ReportPolicyStatusResponse, SandboxLogLine, SandboxPolicyRevision, SettingScope, SettingValue, + SubmitPolicyAnalysisRequest, SubmitPolicyAnalysisResponse, UndoDraftChunkRequest, + UndoDraftChunkResponse, UpdateConfigRequest, UpdateConfigResponse, }; -use openshell_core::proto::{Sandbox, SandboxPolicy as ProtoSandboxPolicy}; -use openshell_core::settings::{self, SettingValueKind}; +use openshell_core::proto::{ + L7DenyRule, L7Rule, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, Provider, Sandbox, + SandboxPolicy as ProtoSandboxPolicy, +}; +use openshell_core::{ + VERSION, + settings::{self, SettingValueKind}, +}; +use openshell_ocsf::{ + ConfigStateChangeBuilder, OCSF_TARGET, OcsfEvent, SandboxContext, SeverityId, StateId, StatusId, +}; +use openshell_policy::{ + PolicyMergeOp, ProviderPolicyLayer, compose_effective_policy, merge_policy, +}; +use openshell_providers::get_default_profile; use prost::Message; use sha2::{Digest, Sha256}; use std::collections::{BTreeMap, HashMap}; +use std::net::{IpAddr, Ipv4Addr}; use std::sync::Arc; use tonic::{Request, Response, Status}; use tracing::{debug, info, warn}; @@ -49,11 +66,9 @@ use super::{MAX_PAGE_SIZE, StoredSettingValue, StoredSettings, clamp_limit, curr /// Internal object type for durable gateway-global settings. const GLOBAL_SETTINGS_OBJECT_TYPE: &str = "gateway_settings"; -/// Internal object id for the singleton global settings record. -const GLOBAL_SETTINGS_ID: &str = "gateway_settings:global"; const GLOBAL_SETTINGS_NAME: &str = "global"; /// Internal object type for durable sandbox-scoped settings. -pub(crate) const SANDBOX_SETTINGS_OBJECT_TYPE: &str = "sandbox_settings"; +pub const SANDBOX_SETTINGS_OBJECT_TYPE: &str = "sandbox_settings"; /// Reserved settings key used to store global policy payload. const POLICY_SETTING_KEY: &str = "policy"; /// Sentinel `sandbox_id` used to store global policy revisions. @@ -61,6 +76,268 @@ const GLOBAL_POLICY_SANDBOX_ID: &str = "__global__"; /// Maximum number of optimistic retry attempts for policy version conflicts. const MERGE_RETRY_LIMIT: usize = 5; +fn emit_gateway_policy_audit_log( + sandbox_id: &str, + sandbox_name: &str, + state_label: &str, + detail: impl Into, + version: i64, + policy_hash: &str, +) { + let message = build_gateway_policy_audit_message( + sandbox_id, + sandbox_name, + state_label, + detail, + version, + policy_hash, + ); + info!( + target: OCSF_TARGET, + sandbox_id = %sandbox_id, + message = %message + ); +} + +fn build_gateway_policy_audit_message( + sandbox_id: &str, + sandbox_name: &str, + state_label: &str, + detail: impl Into, + version: i64, + policy_hash: &str, +) -> String { + let ctx = SandboxContext { + sandbox_id: sandbox_id.to_string(), + sandbox_name: sandbox_name.to_string(), + container_image: "openshell/gateway".to_string(), + hostname: "openshell-gateway".to_string(), + product_version: VERSION.to_string(), + proxy_ip: IpAddr::V4(Ipv4Addr::LOCALHOST), + proxy_port: 0, + }; + let mut builder = ConfigStateChangeBuilder::new(&ctx) + .state(StateId::Other, state_label) + .severity(SeverityId::Informational) + .status(StatusId::Success) + .message(detail.into()); + if version > 0 { + builder = builder.unmapped("policy_version", format!("v{version}")); + } + if !policy_hash.is_empty() { + builder = builder.unmapped("policy_hash", policy_hash.to_string()); + } + let event: OcsfEvent = builder.build(); + event.format_shorthand() +} + +fn summarize_cli_policy_merge_op(operation: &PolicyMergeOp) -> String { + match operation { + PolicyMergeOp::AddRule { rule_name, rule } => summarize_add_endpoint(rule_name, rule), + PolicyMergeOp::RemoveEndpoint { + rule_name, + host, + port, + } => rule_name.as_ref().map_or_else( + || format!("remove-endpoint {host}:{port}"), + |rule_name| format!("remove-endpoint {host}:{port} from rule {rule_name}"), + ), + PolicyMergeOp::RemoveRule { rule_name } => format!("remove-rule {rule_name}"), + PolicyMergeOp::AddDenyRules { + host, + port, + deny_rules, + } => format!( + "add-deny {host}:{port} [{}]", + deny_rules + .iter() + .map(summarize_l7_deny_rule) + .collect::>() + .join(", ") + ), + PolicyMergeOp::AddAllowRules { host, port, rules } => format!( + "add-allow {host}:{port} [{}]", + rules + .iter() + .map(summarize_l7_rule) + .collect::>() + .join(", ") + ), + PolicyMergeOp::RemoveBinary { + rule_name, + binary_path, + } => format!("remove-binary {rule_name} {binary_path}"), + } +} + +fn ensure_chunk_belongs_to_sandbox( + chunk: &DraftChunkRecord, + sandbox_id: &str, +) -> Result<(), Status> { + if chunk.sandbox_id != sandbox_id { + return Err(Status::not_found("chunk not found")); + } + Ok(()) +} + +fn summarize_add_endpoint(rule_name: &str, rule: &NetworkPolicyRule) -> String { + let endpoints = rule + .endpoints + .iter() + .map(summarize_endpoint) + .collect::>() + .join(", "); + let binaries = summarize_binaries(&rule.binaries); + format!("add-endpoint {rule_name} endpoints=[{endpoints}] binaries=[{binaries}]") +} + +fn summarize_add_rule(rule_name: &str, rule: &NetworkPolicyRule) -> String { + let endpoints = rule + .endpoints + .iter() + .map(summarize_endpoint) + .collect::>() + .join(", "); + let binaries = summarize_binaries(&rule.binaries); + format!("add-rule {rule_name} endpoints=[{endpoints}] binaries=[{binaries}]") +} + +fn summarize_endpoint(endpoint: &NetworkEndpoint) -> String { + let mut parts = vec![format!("{}:{}", endpoint.host, endpoint.port)]; + if !endpoint.protocol.is_empty() { + parts.push(format!("protocol={}", endpoint.protocol)); + } + if !endpoint.access.is_empty() { + parts.push(format!("access={}", endpoint.access)); + } + if !endpoint.enforcement.is_empty() { + parts.push(format!("enforcement={}", endpoint.enforcement)); + } + if !endpoint.tls.is_empty() { + parts.push(format!("tls={}", endpoint.tls)); + } + if !endpoint.allowed_ips.is_empty() { + parts.push(format!("allowed_ips={}", endpoint.allowed_ips.len())); + } + if !endpoint.ports.is_empty() { + parts.push(format!("ports={}", endpoint.ports.len())); + } + if !endpoint.rules.is_empty() { + parts.push(format!( + "allow=[{}]", + endpoint + .rules + .iter() + .map(summarize_l7_rule) + .collect::>() + .join(", ") + )); + } + if !endpoint.deny_rules.is_empty() { + parts.push(format!( + "deny=[{}]", + endpoint + .deny_rules + .iter() + .map(summarize_l7_deny_rule) + .collect::>() + .join(", ") + )); + } + parts.join(" ") +} + +fn summarize_l7_rule(rule: &L7Rule) -> String { + let Some(allow) = rule.allow.as_ref() else { + return "allow".to_string(); + }; + summarize_l7_match( + &allow.method, + &allow.path, + &allow.command, + allow.query.len(), + ) +} + +fn summarize_l7_deny_rule(rule: &L7DenyRule) -> String { + summarize_l7_match(&rule.method, &rule.path, &rule.command, rule.query.len()) +} + +fn summarize_l7_match(method: &str, path: &str, command: &str, query_count: usize) -> String { + let mut parts = Vec::new(); + if !method.is_empty() { + parts.push(method.to_string()); + } + if !path.is_empty() { + parts.push(path.to_string()); + } + if !command.is_empty() { + parts.push(format!("command={}", truncate_for_log(command, 48))); + } + if query_count > 0 { + parts.push(format!("query_keys={query_count}")); + } + if parts.is_empty() { + "rule".to_string() + } else { + parts.join(" ") + } +} + +fn summarize_binaries(binaries: &[NetworkBinary]) -> String { + binaries + .iter() + .map(|binary| binary.path.as_str()) + .collect::>() + .join(", ") +} + +fn summarize_draft_chunk_rule(chunk: &DraftChunkRecord) -> Result { + let rule = NetworkPolicyRule::decode(chunk.proposed_rule.as_slice()) + .map_err(|e| Status::internal(format!("decode proposed_rule failed: {e}")))?; + Ok(summarize_add_rule(&chunk.rule_name, &rule)) +} + +fn truncate_for_log(input: &str, max_chars: usize) -> String { + let mut chars = input.chars(); + let truncated: String = chars.by_ref().take(max_chars).collect(); + if chars.next().is_some() { + format!("{truncated}...") + } else { + truncated + } +} + +fn is_sandbox_secret_authenticated(request: &Request) -> bool { + oidc::is_sandbox_secret_authenticated(request.metadata()) +} + +/// Sandbox-secret-authenticated callers may only perform sandbox-scoped policy +/// sync. They must not be able to mutate global config or sandbox settings. +fn validate_sandbox_secret_update(req: &UpdateConfigRequest) -> Result<(), Status> { + if req.global { + return Err(Status::permission_denied( + "sandbox secret cannot mutate global config", + )); + } + if req.delete_setting { + return Err(Status::permission_denied( + "sandbox secret cannot delete settings", + )); + } + if req.name.trim().is_empty() { + return Err(Status::permission_denied( + "sandbox secret may only perform sandbox policy sync", + )); + } + if req.policy.is_none() || !req.setting_key.trim().is_empty() { + return Err(Status::permission_denied( + "sandbox secret may only perform sandbox policy sync", + )); + } + Ok(()) +} + // --------------------------------------------------------------------------- // Config handlers // --------------------------------------------------------------------------- @@ -77,6 +354,11 @@ pub(super) async fn handle_get_sandbox_config( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; + let sandbox_provider_names = sandbox + .spec + .as_ref() + .map(|spec| spec.providers.clone()) + .unwrap_or_default(); // Try to get the latest policy from the policy history table. let latest = state @@ -103,9 +385,10 @@ pub(super) async fn handle_get_sandbox_config( // Lazy backfill: no policy history exists yet. let spec = sandbox .spec + .as_ref() .ok_or_else(|| Status::internal("sandbox has no spec"))?; - match spec.policy { + match spec.policy.clone() { None => { debug!( sandbox_id = %sandbox_id, @@ -151,7 +434,10 @@ pub(super) async fn handle_get_sandbox_config( }; let global_settings = load_global_settings(state.store.as_ref()).await?; - let sandbox_settings = load_sandbox_settings(state.store.as_ref(), &sandbox_id).await?; + let sandbox_settings = + load_sandbox_settings(state.store.as_ref(), sandbox.object_name()).await?; + let providers_v2_enabled = + bool_setting_enabled(&global_settings, settings::PROVIDERS_V2_ENABLED_KEY)?; let mut global_policy_version: u32 = 0; @@ -171,6 +457,19 @@ pub(super) async fn handle_get_sandbox_config( } } + if providers_v2_enabled + && !matches!(policy_source, PolicySource::Global) + && let Some(source_policy) = policy.as_ref() + { + let provider_layers = + profile_provider_policy_layers(state.store.as_ref(), &sandbox_provider_names).await?; + if !provider_layers.is_empty() { + let effective_policy = compose_effective_policy(source_policy, &provider_layers); + policy_hash = deterministic_policy_hash(&effective_policy); + policy = Some(effective_policy); + } + } + let settings = merge_effective_settings(&global_settings, &sandbox_settings)?; let config_revision = compute_config_revision(policy.as_ref(), &settings, policy_source); @@ -185,6 +484,49 @@ pub(super) async fn handle_get_sandbox_config( })) } +async fn profile_provider_policy_layers( + store: &Store, + provider_names: &[String], +) -> Result, Status> { + let mut layers = Vec::new(); + + for name in provider_names { + let provider = store + .get_message_by_name::(name) + .await + .map_err(|e| Status::internal(format!("failed to fetch provider '{name}': {e}")))? + .ok_or_else(|| Status::failed_precondition(format!("provider '{name}' not found")))?; + + let provider_type = provider.r#type.trim(); + let Some(profile) = get_default_profile(provider_type) else { + warn!( + provider_name = %name, + provider_type, + "provider type has no default profile; skipping provider policy layer" + ); + continue; + }; + + let rule_name = openshell_policy::provider_rule_name(provider.object_name()); + layers.push(ProviderPolicyLayer { + rule_name: rule_name.clone(), + rule: profile.network_policy_rule(&rule_name), + }); + } + + Ok(layers) +} + +fn bool_setting_enabled(settings: &StoredSettings, key: &str) -> Result { + match settings.settings.get(key) { + None => Ok(false), + Some(StoredSettingValue::Bool(value)) => Ok(*value), + Some(_) => Err(Status::internal(format!( + "setting '{key}' has invalid value type; expected bool" + ))), + } +} + pub(super) async fn handle_get_gateway_config( state: &Arc, _request: Request, @@ -238,25 +580,40 @@ pub(super) async fn handle_update_config( state: &Arc, request: Request, ) -> Result, Status> { + let sandbox_secret_auth = is_sandbox_secret_authenticated(&request); let req = request.into_inner(); + if sandbox_secret_auth { + validate_sandbox_secret_update(&req)?; + } let key = req.setting_key.trim(); let has_policy = req.policy.is_some(); let has_setting = !key.is_empty(); + let has_merge_ops = !req.merge_operations.is_empty(); + let mut mutation_count = 0_u8; + mutation_count += u8::from(has_policy); + mutation_count += u8::from(has_setting); + mutation_count += u8::from(has_merge_ops); - if has_policy && has_setting { + if mutation_count > 1 { return Err(Status::invalid_argument( - "policy and setting_key cannot be set in the same request", + "policy, setting_key, and merge_operations are mutually exclusive", )); } - if !has_policy && !has_setting { + if mutation_count == 0 { return Err(Status::invalid_argument( - "either policy or setting_key must be provided", + "one of policy, setting_key, or merge_operations must be provided", )); } if req.global { let _settings_guard = state.settings_mutex.lock().await; + if has_merge_ops { + return Err(Status::invalid_argument( + "merge_operations are not supported for global policy updates", + )); + } + if has_policy { if req.delete_setting { return Err(Status::invalid_argument( @@ -417,7 +774,7 @@ pub(super) async fn handle_update_config( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); if has_setting { let _settings_guard = state.settings_mutex.lock().await; @@ -439,14 +796,13 @@ pub(super) async fn handle_update_config( } let mut sandbox_settings = - load_sandbox_settings(state.store.as_ref(), &sandbox_id).await?; + load_sandbox_settings(state.store.as_ref(), sandbox.object_name()).await?; let removed = sandbox_settings.settings.remove(key).is_some(); if removed { sandbox_settings.revision = sandbox_settings.revision.wrapping_add(1); save_sandbox_settings( state.store.as_ref(), - &sandbox_id, - &sandbox.name, + sandbox.object_name(), &sandbox_settings, ) .await?; @@ -472,14 +828,14 @@ pub(super) async fn handle_update_config( .ok_or_else(|| Status::invalid_argument("setting_value is required"))?; let stored = proto_setting_to_stored(key, setting)?; - let mut sandbox_settings = load_sandbox_settings(state.store.as_ref(), &sandbox_id).await?; + let mut sandbox_settings = + load_sandbox_settings(state.store.as_ref(), sandbox.object_name()).await?; let changed = upsert_setting_value(&mut sandbox_settings.settings, key, stored); if changed { sandbox_settings.revision = sandbox_settings.revision.wrapping_add(1); save_sandbox_settings( state.store.as_ref(), - &sandbox_id, - &sandbox.name, + sandbox.object_name(), &sandbox_settings, ) .await?; @@ -493,6 +849,69 @@ pub(super) async fn handle_update_config( })); } + if has_merge_ops { + let global_settings = load_global_settings(state.store.as_ref()).await?; + if global_settings.settings.contains_key(POLICY_SETTING_KEY) { + return Err(Status::failed_precondition( + "policy is managed globally; delete global policy before sandbox policy update", + )); + } + + let spec = sandbox + .spec + .as_ref() + .ok_or_else(|| Status::internal("sandbox has no spec"))?; + let merge_ops = parse_merge_operations(&req.merge_operations)?; + validate_merge_operations_for_server(&merge_ops)?; + let (version, hash) = apply_merge_operations_with_retry( + state.store.as_ref(), + &sandbox_id, + spec.policy.as_ref(), + &merge_ops, + ) + .await?; + + state.sandbox_watch_bus.notify(&sandbox_id); + emit_gateway_policy_audit_log( + &sandbox_id, + sandbox.object_name(), + "merged", + format!( + "gateway merged {} incremental policy operation(s)", + merge_ops.len() + ), + version, + &hash, + ); + for operation in &merge_ops { + emit_gateway_policy_audit_log( + &sandbox_id, + sandbox.object_name(), + "merged", + format!( + "gateway merged incremental policy op: {}", + summarize_cli_policy_merge_op(operation) + ), + version, + &hash, + ); + } + info!( + sandbox_id = %sandbox_id, + version, + policy_hash = %hash, + operation_count = merge_ops.len(), + "UpdateConfig: merged incremental policy operations" + ); + + return Ok(Response::new(UpdateConfigResponse { + version: u32::try_from(version).unwrap_or(0), + policy_hash: hash, + settings_revision: 0, + deleted: false, + })); + } + // Sandbox-scoped policy update. let mut new_policy = req .policy @@ -604,7 +1023,10 @@ pub(super) async fn handle_get_sandbox_policy_status( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - (sandbox.id, sandbox.current_policy_version) + ( + sandbox.object_id().to_string(), + sandbox.current_policy_version, + ) }; let record = if req.version == 0 { @@ -652,7 +1074,7 @@ pub(super) async fn handle_list_sandbox_policies( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - sandbox.id + sandbox.object_id().to_string() }; let limit = clamp_limit(req.limit, 50, MAX_PAGE_SIZE); @@ -839,7 +1261,7 @@ pub(super) async fn handle_submit_policy_analysis( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); let current_version = state .store @@ -954,7 +1376,7 @@ pub(super) async fn handle_get_draft_policy( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); let status_filter = if req.status_filter.is_empty() { None @@ -1016,7 +1438,7 @@ pub(super) async fn handle_approve_draft_chunk( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); let chunk = state .store @@ -1024,6 +1446,7 @@ pub(super) async fn handle_approve_draft_chunk( .await .map_err(|e| Status::internal(format!("fetch chunk failed: {e}")))? .ok_or_else(|| Status::not_found("chunk not found"))?; + ensure_chunk_belongs_to_sandbox(&chunk, &sandbox_id)?; if chunk.status != "pending" && chunk.status != "rejected" { return Err(Status::failed_precondition(format!( @@ -1045,6 +1468,7 @@ pub(super) async fn handle_approve_draft_chunk( let (version, hash) = merge_chunk_into_policy(state.store.as_ref(), &sandbox_id, &chunk).await?; + let chunk_summary = summarize_draft_chunk_rule(&chunk)?; let now_ms = current_time_ms().map_err(|e| Status::internal(format!("timestamp error: {e}")))?; @@ -1055,6 +1479,17 @@ pub(super) async fn handle_approve_draft_chunk( .map_err(|e| Status::internal(format!("update chunk status failed: {e}")))?; state.sandbox_watch_bus.notify(&sandbox_id); + emit_gateway_policy_audit_log( + &sandbox_id, + sandbox.object_name(), + "approved", + format!( + "gateway approved draft chunk {}: {chunk_summary}", + req.chunk_id + ), + version, + &hash, + ); info!( sandbox_id = %sandbox_id, @@ -1089,7 +1524,7 @@ pub(super) async fn handle_reject_draft_chunk( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); let chunk = state .store @@ -1097,6 +1532,7 @@ pub(super) async fn handle_reject_draft_chunk( .await .map_err(|e| Status::internal(format!("fetch chunk failed: {e}")))? .ok_or_else(|| Status::not_found("chunk not found"))?; + ensure_chunk_belongs_to_sandbox(&chunk, &sandbox_id)?; if chunk.status != "pending" && chunk.status != "approved" { return Err(Status::failed_precondition(format!( @@ -1120,7 +1556,18 @@ pub(super) async fn handle_reject_draft_chunk( if was_approved { require_no_global_policy(state).await?; - remove_chunk_from_policy(state, &sandbox_id, &chunk).await?; + let (version, hash) = remove_chunk_from_policy(state, &sandbox_id, &chunk).await?; + emit_gateway_policy_audit_log( + &sandbox_id, + sandbox.object_name(), + "removed", + format!( + "gateway removed previously approved draft chunk {}: remove-binary {} {}", + req.chunk_id, chunk.rule_name, chunk.binary + ), + version, + &hash, + ); } let now_ms = @@ -1153,7 +1600,7 @@ pub(super) async fn handle_approve_all_draft_chunks( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); let pending_chunks = state .store @@ -1203,6 +1650,7 @@ pub(super) async fn handle_approve_all_draft_chunks( merge_chunk_into_policy(state.store.as_ref(), &sandbox_id, chunk).await?; last_version = version; last_hash = hash; + let chunk_summary = summarize_draft_chunk_rule(chunk)?; let now_ms = current_time_ms().map_err(|e| Status::internal(format!("timestamp error: {e}")))?; @@ -1212,10 +1660,28 @@ pub(super) async fn handle_approve_all_draft_chunks( .await .map_err(|e| Status::internal(format!("update chunk status failed: {e}")))?; + emit_gateway_policy_audit_log( + &sandbox_id, + sandbox.object_name(), + "approved", + format!("gateway approved draft chunk {}: {chunk_summary}", chunk.id), + version, + &last_hash, + ); chunks_approved += 1; } state.sandbox_watch_bus.notify(&sandbox_id); + emit_gateway_policy_audit_log( + &sandbox_id, + sandbox.object_name(), + "merged", + format!( + "gateway bulk-approved {chunks_approved} draft chunk(s) and skipped {chunks_skipped}" + ), + last_version, + &last_hash, + ); info!( sandbox_id = %sandbox_id, @@ -1249,12 +1715,13 @@ pub(super) async fn handle_edit_draft_chunk( .proposed_rule .ok_or_else(|| Status::invalid_argument("proposed_rule is required"))?; - let _sandbox = state + let sandbox = state .store .get_message_by_name::(&req.name) .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; + let sandbox_id = sandbox.object_id().to_string(); let chunk = state .store @@ -1262,6 +1729,7 @@ pub(super) async fn handle_edit_draft_chunk( .await .map_err(|e| Status::internal(format!("fetch chunk failed: {e}")))? .ok_or_else(|| Status::not_found("chunk not found"))?; + ensure_chunk_belongs_to_sandbox(&chunk, &sandbox_id)?; if chunk.status != "pending" { return Err(Status::failed_precondition(format!( @@ -1303,7 +1771,7 @@ pub(super) async fn handle_undo_draft_chunk( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); let chunk = state .store @@ -1311,6 +1779,7 @@ pub(super) async fn handle_undo_draft_chunk( .await .map_err(|e| Status::internal(format!("fetch chunk failed: {e}")))? .ok_or_else(|| Status::not_found("chunk not found"))?; + ensure_chunk_belongs_to_sandbox(&chunk, &sandbox_id)?; if chunk.status != "approved" { return Err(Status::failed_precondition(format!( @@ -1337,6 +1806,17 @@ pub(super) async fn handle_undo_draft_chunk( .map_err(|e| Status::internal(format!("update chunk status failed: {e}")))?; state.sandbox_watch_bus.notify(&sandbox_id); + emit_gateway_policy_audit_log( + &sandbox_id, + sandbox.object_name(), + "removed", + format!( + "gateway reverted approved draft chunk {}: remove-binary {} {}", + req.chunk_id, chunk.rule_name, chunk.binary + ), + version, + &hash, + ); info!( sandbox_id = %sandbox_id, @@ -1368,7 +1848,7 @@ pub(super) async fn handle_clear_draft_chunks( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); let deleted = state .store @@ -1404,7 +1884,7 @@ pub(super) async fn handle_get_draft_history( .await .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? .ok_or_else(|| Status::not_found("sandbox not found"))?; - let sandbox_id = sandbox.id.clone(); + let sandbox_id = sandbox.object_id().to_string(); let all_chunks = state .store @@ -1615,23 +2095,21 @@ fn generate_security_notes(host: &str, port: u16) -> String { /// /// This is defense-in-depth: the proxy blocks these at runtime, so /// merging them into the active policy would be silently un-enforceable. -fn validate_rule_not_always_blocked( - rule: &openshell_core::proto::NetworkPolicyRule, -) -> Result<(), Status> { +fn validate_rule_not_always_blocked(rule: &NetworkPolicyRule) -> Result<(), Status> { use openshell_core::net::{is_always_blocked_ip, is_always_blocked_net}; use std::net::IpAddr; for ep in &rule.endpoints { // Check if the endpoint host is a literal always-blocked IP. - if let Ok(ip) = ep.host.parse::() { - if is_always_blocked_ip(ip) { - return Err(Status::invalid_argument(format!( - "proposed rule endpoint host '{}' is an always-blocked address \ - (loopback/link-local/unspecified); the proxy will deny traffic \ - to this destination regardless of policy", - ep.host - ))); - } + if let Ok(ip) = ep.host.parse::() + && is_always_blocked_ip(ip) + { + return Err(Status::invalid_argument(format!( + "proposed rule endpoint host '{}' is an always-blocked address \ + (loopback/link-local/unspecified); the proxy will deny traffic \ + to this destination regardless of policy", + ep.host + ))); } let host_lc = ep.host.to_lowercase(); if host_lc == "localhost" || host_lc == "localhost." { @@ -1650,14 +2128,14 @@ fn validate_rule_not_always_blocked( IpAddr::V6(v6) => ipnet::IpNet::V6(ipnet::Ipv6Net::from(v6)), }) }); - if let Ok(net) = parsed { - if is_always_blocked_net(net) { - return Err(Status::invalid_argument(format!( - "proposed rule contains always-blocked allowed_ips entry '{entry}'; \ - SSRF hardening prevents traffic to these destinations \ - regardless of policy" - ))); - } + if let Ok(net) = parsed + && is_always_blocked_net(net) + { + return Err(Status::invalid_argument(format!( + "proposed rule contains always-blocked `allowed_ips` entry '{entry}'; \ + SSRF hardening prevents traffic to these destinations \ + regardless of policy" + ))); } // Invalid entries are not our concern here — the sandbox's // parse_allowed_ips handles syntax validation. @@ -1677,90 +2155,203 @@ async fn require_no_global_policy(state: &ServerState) -> Result<(), Status> { Ok(()) } -pub(super) async fn merge_chunk_into_policy( - store: &Store, - sandbox_id: &str, - chunk: &DraftChunkRecord, -) -> Result<(i64, String), Status> { - use openshell_core::proto::NetworkPolicyRule; +fn parse_merge_operations( + proto_ops: &[PolicyMergeOperation], +) -> Result, Status> { + proto_ops + .iter() + .enumerate() + .map(|(index, operation)| { + let Some(operation) = operation.operation.as_ref() else { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}] is missing an operation" + ))); + }; - let rule = NetworkPolicyRule::decode(chunk.proposed_rule.as_slice()) - .map_err(|e| Status::internal(format!("decode proposed_rule failed: {e}")))?; + match operation { + policy_merge_operation::Operation::AddRule(add_rule) => { + let rule_name = add_rule.rule_name.trim(); + if rule_name.is_empty() { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}].add_rule.rule_name is required" + ))); + } + if add_rule.rule.as_ref().is_none_or(|rule| rule.endpoints.is_empty()) { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}].add_rule.rule must contain at least one endpoint" + ))); + } + Ok(PolicyMergeOp::AddRule { + rule_name: rule_name.to_string(), + rule: add_rule.rule.clone().unwrap_or_default(), + }) + } + policy_merge_operation::Operation::RemoveEndpoint(remove_endpoint) => { + if remove_endpoint.host.trim().is_empty() || remove_endpoint.port == 0 { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}].remove_endpoint requires host and non-zero port" + ))); + } + let rule_name = if remove_endpoint.rule_name.trim().is_empty() { + None + } else { + Some(remove_endpoint.rule_name.trim().to_string()) + }; + Ok(PolicyMergeOp::RemoveEndpoint { + rule_name, + host: remove_endpoint.host.trim().to_string(), + port: remove_endpoint.port, + }) + } + policy_merge_operation::Operation::RemoveRule(remove_rule) => { + let rule_name = remove_rule.rule_name.trim(); + if rule_name.is_empty() { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}].remove_rule.rule_name is required" + ))); + } + Ok(PolicyMergeOp::RemoveRule { + rule_name: rule_name.to_string(), + }) + } + policy_merge_operation::Operation::AddDenyRules(add_deny_rules) => { + parse_proto_add_deny_rules(index, add_deny_rules) + } + policy_merge_operation::Operation::AddAllowRules(add_allow_rules) => { + parse_proto_add_allow_rules(index, add_allow_rules) + } + policy_merge_operation::Operation::RemoveBinary(remove_binary) => { + let rule_name = remove_binary.rule_name.trim(); + let binary_path = remove_binary.binary_path.trim(); + if rule_name.is_empty() || binary_path.is_empty() { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}].remove_binary requires rule_name and binary_path" + ))); + } + Ok(PolicyMergeOp::RemoveBinary { + rule_name: rule_name.to_string(), + binary_path: binary_path.to_string(), + }) + } + } + }) + .collect() +} + +fn parse_proto_add_deny_rules( + index: usize, + add_deny_rules: &ProtoAddDenyRules, +) -> Result { + if add_deny_rules.host.trim().is_empty() + || add_deny_rules.port == 0 + || add_deny_rules.deny_rules.is_empty() + { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}].add_deny_rules requires host, non-zero port, and at least one deny rule" + ))); + } + + Ok(PolicyMergeOp::AddDenyRules { + host: add_deny_rules.host.trim().to_string(), + port: add_deny_rules.port, + deny_rules: add_deny_rules.deny_rules.clone(), + }) +} + +fn parse_proto_add_allow_rules( + index: usize, + add_allow_rules: &ProtoAddAllowRules, +) -> Result { + if add_allow_rules.host.trim().is_empty() + || add_allow_rules.port == 0 + || add_allow_rules.rules.is_empty() + { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}].add_allow_rules requires host, non-zero port, and at least one allow rule" + ))); + } + if add_allow_rules + .rules + .iter() + .any(|rule| rule.allow.as_ref().is_none()) + { + return Err(Status::invalid_argument(format!( + "merge_operations[{index}].add_allow_rules rules must include allow payloads" + ))); + } + + Ok(PolicyMergeOp::AddAllowRules { + host: add_allow_rules.host.trim().to_string(), + port: add_allow_rules.port, + rules: add_allow_rules.rules.clone(), + }) +} + +fn validate_merge_operations_for_server(operations: &[PolicyMergeOp]) -> Result<(), Status> { + for operation in operations { + if let PolicyMergeOp::AddRule { rule, .. } = operation { + validate_rule_not_always_blocked(rule)?; + } + } + Ok(()) +} - // Defense-in-depth: reject proposed rules targeting always-blocked - // destinations. Even if the sandbox mapper didn't filter these (e.g., - // an older sandbox version), the proxy will deny them at runtime. - validate_rule_not_always_blocked(&rule)?; +fn map_policy_merge_error(error: openshell_policy::PolicyMergeError) -> Status { + match error { + openshell_policy::PolicyMergeError::MissingRuleNameForAddRule + | openshell_policy::PolicyMergeError::InvalidEndpointReference { .. } + | openshell_policy::PolicyMergeError::UnsupportedAccessPreset { .. } => { + Status::invalid_argument(error.to_string()) + } + openshell_policy::PolicyMergeError::EndpointNotFound { .. } + | openshell_policy::PolicyMergeError::EndpointHasNoL7Inspection { .. } + | openshell_policy::PolicyMergeError::UnsupportedEndpointProtocol { .. } + | openshell_policy::PolicyMergeError::EndpointHasNoAllowBase { .. } => { + Status::failed_precondition(error.to_string()) + } + } +} +async fn apply_merge_operations_with_retry( + store: &Store, + sandbox_id: &str, + baseline_policy: Option<&ProtoSandboxPolicy>, + operations: &[PolicyMergeOp], +) -> Result<(i64, String), Status> { for attempt in 1..=MERGE_RETRY_LIMIT { let latest = store .get_latest_policy(sandbox_id) .await .map_err(|e| Status::internal(format!("fetch latest policy failed: {e}")))?; - let mut policy = if let Some(ref record) = latest { + let current_policy = if let Some(ref record) = latest { ProtoSandboxPolicy::decode(record.policy_payload.as_slice()) .map_err(|e| Status::internal(format!("decode current policy failed: {e}")))? } else { - ProtoSandboxPolicy::default() + baseline_policy.cloned().unwrap_or_default() }; - let base_version = latest.as_ref().map_or(0, |r| r.version); + let merged = merge_policy(current_policy, operations).map_err(map_policy_merge_error)?; + let new_policy = merged.policy; + let hash = deterministic_policy_hash(&new_policy); - let chunk_host_lc = chunk.host.to_lowercase(); - let chunk_port = chunk.port as u32; + if let Some(baseline_policy) = baseline_policy { + validate_static_fields_unchanged(baseline_policy, &new_policy)?; + } + validate_policy_safety(&new_policy)?; - let merge_key = if policy.network_policies.contains_key(&chunk.rule_name) { - Some(chunk.rule_name.clone()) - } else { - policy - .network_policies - .iter() - .find_map(|(key, existing_rule)| { - let has_match = existing_rule.endpoints.iter().any(|ep| { - let host_match = ep.host.to_lowercase() == chunk_host_lc; - let port_match = if ep.ports.is_empty() { - ep.port == chunk_port - } else { - ep.ports.contains(&chunk_port) - }; - host_match && port_match - }); - has_match.then(|| key.clone()) - }) - }; + if let Some(ref current) = latest + && current.policy_hash == hash + { + return Ok((current.version, hash)); + } - if let Some(key) = merge_key { - let existing = policy.network_policies.get_mut(&key).unwrap(); - for b in &rule.binaries { - if !existing.binaries.iter().any(|eb| eb.path == b.path) { - existing.binaries.push(b.clone()); - } - } - for ep in &rule.endpoints { - if let Some(existing_ep) = existing.endpoints.iter_mut().find(|e| { - e.host.to_lowercase() == ep.host.to_lowercase() - && (e.port == ep.port - || (!e.ports.is_empty() && e.ports.contains(&ep.port))) - }) { - for ip in &ep.allowed_ips { - if !existing_ep.allowed_ips.contains(ip) { - existing_ep.allowed_ips.push(ip.clone()); - } - } - } else { - existing.endpoints.push(ep.clone()); - } - } - } else { - policy - .network_policies - .insert(chunk.rule_name.clone(), rule.clone()); + if latest.is_none() && !merged.changed { + return Ok((0, hash)); } - let payload = policy.encode_to_vec(); - let hash = deterministic_policy_hash(&policy); - let next_version = base_version + 1; + let payload = new_policy.encode_to_vec(); + let next_version = latest.as_ref().map_or(1, |record| record.version + 1); let policy_id = uuid::Uuid::new_v4().to_string(); match store @@ -1775,24 +2366,23 @@ pub(super) async fn merge_chunk_into_policy( if attempt > 1 { info!( sandbox_id = %sandbox_id, - rule_name = %chunk.rule_name, attempt, version = next_version, - "merge_chunk_into_policy: succeeded after version conflict retry" + operation_count = operations.len(), + "apply_merge_operations_with_retry: succeeded after version conflict retry" ); } return Ok((next_version, hash)); } Err(e) => { - let msg = e.to_string(); - if msg.contains("UNIQUE") || msg.contains("unique") || msg.contains("duplicate") { + if e.is_unique_violation_on("objects_version_uq") { warn!( sandbox_id = %sandbox_id, - rule_name = %chunk.rule_name, attempt, conflicting_version = next_version, - "merge_chunk_into_policy: version conflict, retrying" + operation_count = operations.len(), + "apply_merge_operations_with_retry: version conflict, retrying" ); tokio::task::yield_now().await; continue; @@ -1805,90 +2395,44 @@ pub(super) async fn merge_chunk_into_policy( } Err(Status::aborted(format!( - "merge_chunk_into_policy: gave up after {} version conflict retries for rule '{}'", - MERGE_RETRY_LIMIT, chunk.rule_name + "apply_merge_operations_with_retry: gave up after {MERGE_RETRY_LIMIT} version conflict retries" ))) } -async fn remove_chunk_from_policy( - state: &ServerState, +pub(super) async fn merge_chunk_into_policy( + store: &Store, sandbox_id: &str, chunk: &DraftChunkRecord, ) -> Result<(i64, String), Status> { - for attempt in 1..=MERGE_RETRY_LIMIT { - let latest = state - .store - .get_latest_policy(sandbox_id) - .await - .map_err(|e| Status::internal(format!("fetch latest policy failed: {e}")))? - .ok_or_else(|| Status::internal("no active policy to undo from"))?; - - let mut policy = ProtoSandboxPolicy::decode(latest.policy_payload.as_slice()) - .map_err(|e| Status::internal(format!("decode current policy failed: {e}")))?; - - let should_remove = - if let Some(existing) = policy.network_policies.get_mut(&chunk.rule_name) { - existing.binaries.retain(|b| b.path != chunk.binary); - existing.binaries.is_empty() - } else { - false - }; - if should_remove { - policy.network_policies.remove(&chunk.rule_name); - } - - let payload = policy.encode_to_vec(); - let hash = deterministic_policy_hash(&policy); - let next_version = latest.version + 1; - let policy_id = uuid::Uuid::new_v4().to_string(); - - match state - .store - .put_policy_revision(&policy_id, sandbox_id, next_version, &payload, &hash) - .await - { - Ok(()) => { - let _ = state - .store - .supersede_older_policies(sandbox_id, next_version) - .await; - - if attempt > 1 { - info!( - sandbox_id = %sandbox_id, - rule_name = %chunk.rule_name, - attempt, - version = next_version, - "remove_chunk_from_policy: succeeded after version conflict retry" - ); - } - - return Ok((next_version, hash)); - } - Err(e) => { - let msg = e.to_string(); - if msg.contains("UNIQUE") || msg.contains("unique") || msg.contains("duplicate") { - warn!( - sandbox_id = %sandbox_id, - rule_name = %chunk.rule_name, - attempt, - conflicting_version = next_version, - "remove_chunk_from_policy: version conflict, retrying" - ); - tokio::task::yield_now().await; - continue; - } - return Err(Status::internal(format!( - "persist policy revision failed: {e}" - ))); - } - } - } + let rule = NetworkPolicyRule::decode(chunk.proposed_rule.as_slice()) + .map_err(|e| Status::internal(format!("decode proposed_rule failed: {e}")))?; + apply_merge_operations_with_retry( + store, + sandbox_id, + None, + &[PolicyMergeOp::AddRule { + rule_name: chunk.rule_name.clone(), + rule, + }], + ) + .await +} - Err(Status::aborted(format!( - "remove_chunk_from_policy: gave up after {} version conflict retries for rule '{}'", - MERGE_RETRY_LIMIT, chunk.rule_name - ))) +async fn remove_chunk_from_policy( + state: &ServerState, + sandbox_id: &str, + chunk: &DraftChunkRecord, +) -> Result<(i64, String), Status> { + apply_merge_operations_with_retry( + state.store.as_ref(), + sandbox_id, + None, + &[PolicyMergeOp::RemoveBinary { + rule_name: chunk.rule_name.clone(), + binary_path: chunk.binary.clone(), + }], + ) + .await } // --------------------------------------------------------------------------- @@ -1973,7 +2517,7 @@ fn upsert_setting_value( } pub(super) async fn load_global_settings(store: &Store) -> Result { - load_settings_record(store, GLOBAL_SETTINGS_OBJECT_TYPE, GLOBAL_SETTINGS_ID).await + load_settings_record(store, GLOBAL_SETTINGS_OBJECT_TYPE, GLOBAL_SETTINGS_NAME).await } pub(super) async fn save_global_settings( @@ -1983,53 +2527,34 @@ pub(super) async fn save_global_settings( save_settings_record( store, GLOBAL_SETTINGS_OBJECT_TYPE, - GLOBAL_SETTINGS_ID, GLOBAL_SETTINGS_NAME, settings, ) .await } -/// Derive a distinct settings record ID from a sandbox UUID. -pub(crate) fn sandbox_settings_id(sandbox_id: &str) -> String { - format!("settings:{sandbox_id}") -} - pub(super) async fn load_sandbox_settings( store: &Store, - sandbox_id: &str, + sandbox_name: &str, ) -> Result { - load_settings_record( - store, - SANDBOX_SETTINGS_OBJECT_TYPE, - &sandbox_settings_id(sandbox_id), - ) - .await + load_settings_record(store, SANDBOX_SETTINGS_OBJECT_TYPE, sandbox_name).await } pub(super) async fn save_sandbox_settings( store: &Store, - sandbox_id: &str, sandbox_name: &str, settings: &StoredSettings, ) -> Result<(), Status> { - save_settings_record( - store, - SANDBOX_SETTINGS_OBJECT_TYPE, - &sandbox_settings_id(sandbox_id), - sandbox_name, - settings, - ) - .await + save_settings_record(store, SANDBOX_SETTINGS_OBJECT_TYPE, sandbox_name, settings).await } async fn load_settings_record( store: &Store, object_type: &str, - id: &str, + name: &str, ) -> Result { let record = store - .get(object_type, id) + .get_by_name(object_type, name) .await .map_err(|e| Status::internal(format!("fetch settings failed: {e}")))?; if let Some(record) = record { @@ -2043,14 +2568,19 @@ async fn load_settings_record( async fn save_settings_record( store: &Store, object_type: &str, - id: &str, name: &str, settings: &StoredSettings, ) -> Result<(), Status> { let payload = serde_json::to_vec(settings) .map_err(|e| Status::internal(format!("encode settings payload failed: {e}")))?; store - .put(object_type, id, name, &payload) + .put( + object_type, + &uuid::Uuid::new_v4().to_string(), + name, + &payload, + None, + ) .await .map_err(|e| Status::internal(format!("persist settings failed: {e}")))?; Ok(()) @@ -2149,10 +2679,61 @@ fn materialize_global_settings( #[cfg(test)] mod tests { use super::*; + use crate::ServerState; + use crate::compute::new_test_runtime; use crate::persistence::Store; + use crate::sandbox_index::SandboxIndex; + use crate::sandbox_watch::SandboxWatchBus; + use crate::supervisor_session::SupervisorSessionRegistry; + use crate::tracing_bus::TracingLogBus; + use openshell_core::Config; use std::collections::HashMap; + use std::sync::Arc; use tonic::Code; + #[test] + fn sandbox_secret_update_validation_allows_sandbox_policy_sync() { + let req = UpdateConfigRequest { + name: "sandbox-1".to_string(), + policy: Some(ProtoSandboxPolicy::default()), + ..Default::default() + }; + assert!(validate_sandbox_secret_update(&req).is_ok()); + } + + #[test] + fn sandbox_secret_update_validation_rejects_global_mutation() { + let req = UpdateConfigRequest { + global: true, + policy: Some(ProtoSandboxPolicy::default()), + ..Default::default() + }; + let err = validate_sandbox_secret_update(&req).unwrap_err(); + assert_eq!(err.code(), Code::PermissionDenied); + } + + #[test] + fn sandbox_secret_update_validation_rejects_setting_mutation() { + let req = UpdateConfigRequest { + name: "sandbox-1".to_string(), + setting_key: "inference.model".to_string(), + setting_value: Some(SettingValue { value: None }), + ..Default::default() + }; + let err = validate_sandbox_secret_update(&req).unwrap_err(); + assert_eq!(err.code(), Code::PermissionDenied); + } + + #[test] + fn sandbox_secret_marker_detected_from_metadata() { + let mut req = Request::new(()); + req.metadata_mut().insert( + oidc::INTERNAL_AUTH_SOURCE_HEADER, + oidc::AUTH_SOURCE_SANDBOX_SECRET.parse().unwrap(), + ); + assert!(is_sandbox_secret_authenticated(&req)); + } + // ---- Sandbox without policy ---- #[tokio::test] @@ -2162,38 +2743,550 @@ mod tests { let store = Store::connect("sqlite::memory:").await.unwrap(); let sandbox = Sandbox { - id: "sb-no-policy".to_string(), - name: "no-policy-sandbox".to_string(), - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "sb-no-policy".to_string(), + name: "no-policy-sandbox".to_string(), + created_at_ms: 1_000_000, + labels: std::collections::HashMap::new(), + }), + spec: Some(SandboxSpec { + policy: None, + ..Default::default() + }), + phase: SandboxPhase::Provisioning as i32, + ..Default::default() + }; + store.put_message(&sandbox).await.unwrap(); + + let loaded = store + .get_message::("sb-no-policy") + .await + .unwrap() + .unwrap(); + assert!(loaded.spec.unwrap().policy.is_none()); + } + + fn test_provider(name: &str, provider_type: &str) -> Provider { + Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("provider-{name}"), + name: name.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), + r#type: provider_type.to_string(), + credentials: std::iter::once(("GITHUB_TOKEN".to_string(), "ghp-test".to_string())) + .collect(), + config: HashMap::new(), + } + } + + fn test_policy_with_rule(rule_name: &str, host: &str) -> ProtoSandboxPolicy { + ProtoSandboxPolicy { + network_policies: std::iter::once(( + rule_name.to_string(), + NetworkPolicyRule { + name: rule_name.to_string(), + endpoints: vec![NetworkEndpoint { + host: host.to_string(), + port: 443, + ..Default::default() + }], + ..Default::default() + }, + )) + .collect(), + ..Default::default() + } + } + + fn test_sandbox( + id: &str, + name: &str, + policy: ProtoSandboxPolicy, + providers: Vec, + ) -> Sandbox { + use openshell_core::proto::{SandboxPhase, SandboxSpec}; + + Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: id.to_string(), + name: name.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), + spec: Some(SandboxSpec { + policy: Some(policy), + providers, + ..Default::default() + }), + phase: SandboxPhase::Ready as i32, + ..Default::default() + } + } + + async fn enable_providers_v2(state: &Arc) { + let global_settings = StoredSettings { + revision: 1, + settings: std::iter::once(( + settings::PROVIDERS_V2_ENABLED_KEY.to_string(), + StoredSettingValue::Bool(true), + )) + .collect(), + }; + save_global_settings(state.store.as_ref(), &global_settings) + .await + .unwrap(); + } + + async fn get_sandbox_policy(state: &Arc, sandbox_id: &str) -> ProtoSandboxPolicy { + handle_get_sandbox_config( + state, + Request::new(GetSandboxConfigRequest { + sandbox_id: sandbox_id.to_string(), + }), + ) + .await + .unwrap() + .into_inner() + .policy + .expect("sandbox config should include policy") + } + + #[tokio::test] + async fn provider_policy_layers_skip_unknown_provider_types() { + let store = Store::connect("sqlite::memory:").await.unwrap(); + store + .put_message(&test_provider("custom-provider", "custom")) + .await + .unwrap(); + + let layers = profile_provider_policy_layers(&store, &["custom-provider".to_string()]) + .await + .unwrap(); + + assert!(layers.is_empty()); + } + + #[tokio::test] + async fn provider_policy_layers_include_known_provider_profiles() { + let store = Store::connect("sqlite::memory:").await.unwrap(); + store + .put_message(&test_provider("work-github", "github")) + .await + .unwrap(); + + let layers = profile_provider_policy_layers(&store, &["work-github".to_string()]) + .await + .unwrap(); + + assert_eq!(layers.len(), 1); + assert_eq!(layers[0].rule_name, "_provider_work_github"); + assert_eq!(layers[0].rule.endpoints.len(), 2); + assert!( + layers[0] + .rule + .endpoints + .iter() + .any(|endpoint| endpoint.host == "api.github.com") + ); + } + + #[test] + fn providers_v2_enabled_defaults_false_when_unset() { + assert!( + !bool_setting_enabled( + &StoredSettings::default(), + settings::PROVIDERS_V2_ENABLED_KEY + ) + .unwrap() + ); + } + + #[test] + fn providers_v2_enabled_reads_global_bool_setting() { + let mut settings = StoredSettings::default(); + settings.settings.insert( + settings::PROVIDERS_V2_ENABLED_KEY.to_string(), + StoredSettingValue::Bool(true), + ); + + assert!(bool_setting_enabled(&settings, settings::PROVIDERS_V2_ENABLED_KEY).unwrap()); + } + + #[tokio::test] + async fn sandbox_config_omits_provider_layers_when_v2_disabled() { + let state = test_server_state().await; + state + .store + .put_message(&test_provider("work-github", "github")) + .await + .unwrap(); + state + .store + .put_message(&test_sandbox( + "sb-v2-disabled", + "v2-disabled", + test_policy_with_rule("sandbox_only", "sandbox.example.com"), + vec!["work-github".to_string()], + )) + .await + .unwrap(); + + let effective_policy = get_sandbox_policy(&state, "sb-v2-disabled").await; + + assert!( + effective_policy + .network_policies + .contains_key("sandbox_only") + ); + assert!( + !effective_policy + .network_policies + .contains_key("_provider_work_github") + ); + } + + #[tokio::test] + async fn sandbox_config_composes_provider_layers_when_v2_enabled() { + let state = test_server_state().await; + enable_providers_v2(&state).await; + state + .store + .put_message(&test_provider("work-github", "github")) + .await + .unwrap(); + state + .store + .put_message(&test_sandbox( + "sb-v2-enabled", + "v2-enabled", + test_policy_with_rule("sandbox_only", "sandbox.example.com"), + vec!["work-github".to_string()], + )) + .await + .unwrap(); + + let effective_policy = get_sandbox_policy(&state, "sb-v2-enabled").await; + + assert!( + effective_policy + .network_policies + .contains_key("sandbox_only") + ); + assert!( + effective_policy + .network_policies + .contains_key("_provider_work_github") + ); + assert!( + effective_policy + .network_policies + .get("_provider_work_github") + .unwrap() + .endpoints + .iter() + .any(|endpoint| endpoint.host == "api.github.com") + ); + } + + #[tokio::test] + async fn sandbox_config_skips_profileless_provider_types_when_v2_enabled() { + let state = test_server_state().await; + enable_providers_v2(&state).await; + state + .store + .put_message(&test_provider("legacy-generic", "generic")) + .await + .unwrap(); + state + .store + .put_message(&test_provider("custom-provider", "custom")) + .await + .unwrap(); + state + .store + .put_message(&test_sandbox( + "sb-profileless", + "profileless", + test_policy_with_rule("sandbox_only", "sandbox.example.com"), + vec!["legacy-generic".to_string(), "custom-provider".to_string()], + )) + .await + .unwrap(); + + let effective_policy = get_sandbox_policy(&state, "sb-profileless").await; + + assert_eq!(effective_policy.network_policies.len(), 1); + assert!( + effective_policy + .network_policies + .contains_key("sandbox_only") + ); + } + + #[tokio::test] + async fn sandbox_config_composition_is_jit_and_does_not_persist_provider_layers() { + let state = test_server_state().await; + enable_providers_v2(&state).await; + state + .store + .put_message(&test_provider("work-github", "github")) + .await + .unwrap(); + state + .store + .put_message(&test_sandbox( + "sb-jit", + "jit", + test_policy_with_rule("sandbox_only", "sandbox.example.com"), + vec!["work-github".to_string()], + )) + .await + .unwrap(); + + let effective_policy = get_sandbox_policy(&state, "sb-jit").await; + assert!( + effective_policy + .network_policies + .contains_key("_provider_work_github") + ); + + let persisted = state + .store + .get_latest_policy("sb-jit") + .await + .unwrap() + .expect("sandbox policy should be lazily backfilled"); + let persisted_policy = ProtoSandboxPolicy::decode(persisted.policy_payload.as_slice()) + .expect("persisted sandbox policy should decode"); + assert!( + persisted_policy + .network_policies + .contains_key("sandbox_only") + ); + assert!( + !persisted_policy + .network_policies + .contains_key("_provider_work_github") + ); + } + + #[tokio::test] + async fn sandbox_config_preserves_overlapping_user_and_provider_rules() { + let state = test_server_state().await; + enable_providers_v2(&state).await; + state + .store + .put_message(&test_provider("work-github", "github")) + .await + .unwrap(); + state + .store + .put_message(&test_sandbox( + "sb-overlap", + "overlap", + test_policy_with_rule("_provider_work_github", "api.github.com"), + vec!["work-github".to_string()], + )) + .await + .unwrap(); + + let effective_policy = get_sandbox_policy(&state, "sb-overlap").await; + + assert!( + effective_policy + .network_policies + .contains_key("_provider_work_github") + ); + assert!( + effective_policy + .network_policies + .contains_key("_provider_work_github_2") + ); + assert_eq!( + effective_policy + .network_policies + .get("_provider_work_github") + .unwrap() + .endpoints[0] + .host, + "api.github.com" + ); + } + + #[tokio::test] + async fn provider_environment_resolution_is_unchanged_by_providers_v2_setting() { + use openshell_core::proto::GetSandboxProviderEnvironmentRequest; + + let state = test_server_state().await; + state + .store + .put_message(&test_provider("work-github", "github")) + .await + .unwrap(); + state + .store + .put_message(&test_sandbox( + "sb-provider-env", + "provider-env", + test_policy_with_rule("sandbox_only", "sandbox.example.com"), + vec!["work-github".to_string()], + )) + .await + .unwrap(); + + let legacy_env = handle_get_sandbox_provider_environment( + &state, + Request::new(GetSandboxProviderEnvironmentRequest { + sandbox_id: "sb-provider-env".to_string(), + }), + ) + .await + .unwrap() + .into_inner() + .environment; + + enable_providers_v2(&state).await; + let v2_env = handle_get_sandbox_provider_environment( + &state, + Request::new(GetSandboxProviderEnvironmentRequest { + sandbox_id: "sb-provider-env".to_string(), + }), + ) + .await + .unwrap() + .into_inner() + .environment; + + assert_eq!(legacy_env, v2_env); + assert_eq!(v2_env.get("GITHUB_TOKEN"), Some(&"ghp-test".to_string())); + } + + #[tokio::test] + async fn global_policy_suppresses_provider_profile_layers_when_v2_enabled() { + use openshell_core::proto::{ + GetSandboxConfigRequest, NetworkEndpoint, NetworkPolicyRule, SandboxPhase, + SandboxPolicy, SandboxSpec, + }; + + let state = test_server_state().await; + state + .store + .put_message(&test_provider("work-github", "github")) + .await + .unwrap(); + + let sandbox_policy = SandboxPolicy { + network_policies: std::iter::once(( + "sandbox_only".to_string(), + NetworkPolicyRule { + name: "sandbox_only".to_string(), + endpoints: vec![NetworkEndpoint { + host: "sandbox.example.com".to_string(), + port: 443, + ..Default::default() + }], + ..Default::default() + }, + )) + .collect(), + ..Default::default() + }; + let sandbox = Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "sb-global-profile".to_string(), + name: "global-profile-sandbox".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), spec: Some(SandboxSpec { - policy: None, + policy: Some(sandbox_policy), + providers: vec!["work-github".to_string()], ..Default::default() }), - phase: SandboxPhase::Provisioning as i32, + phase: SandboxPhase::Ready as i32, ..Default::default() }; - store.put_message(&sandbox).await.unwrap(); + state.store.put_message(&sandbox).await.unwrap(); - let loaded = store - .get_message::("sb-no-policy") + let global_policy = SandboxPolicy { + network_policies: std::iter::once(( + "global_only".to_string(), + NetworkPolicyRule { + name: "global_only".to_string(), + endpoints: vec![NetworkEndpoint { + host: "global.example.com".to_string(), + port: 443, + ..Default::default() + }], + ..Default::default() + }, + )) + .collect(), + ..Default::default() + }; + let global_settings = StoredSettings { + revision: 1, + settings: [ + ( + settings::PROVIDERS_V2_ENABLED_KEY.to_string(), + StoredSettingValue::Bool(true), + ), + ( + POLICY_SETTING_KEY.to_string(), + StoredSettingValue::Bytes(hex::encode(global_policy.encode_to_vec())), + ), + ] + .into_iter() + .collect(), + }; + save_global_settings(state.store.as_ref(), &global_settings) .await - .unwrap() .unwrap(); - assert!(loaded.spec.unwrap().policy.is_none()); + + let response = handle_get_sandbox_config( + &state, + Request::new(GetSandboxConfigRequest { + sandbox_id: "sb-global-profile".to_string(), + }), + ) + .await + .unwrap() + .into_inner(); + + let effective_policy = response.policy.expect("global policy should be returned"); + assert_eq!(response.policy_source, PolicySource::Global as i32); + assert!( + effective_policy + .network_policies + .contains_key("global_only") + ); + assert!( + !effective_policy + .network_policies + .contains_key("sandbox_only") + ); + assert!( + !effective_policy + .network_policies + .contains_key("_provider_work_github") + ); } #[tokio::test] async fn sandbox_policy_backfill_on_update_when_no_baseline() { - use openshell_core::proto::{ - FilesystemPolicy, LandlockPolicy, ProcessPolicy, SandboxPhase, SandboxSpec, - }; + use openshell_core::proto::{FilesystemPolicy, LandlockPolicy, SandboxPhase, SandboxSpec}; let store = Store::connect("sqlite::memory:").await.unwrap(); let sandbox = Sandbox { - id: "sb-backfill".to_string(), - name: "backfill-sandbox".to_string(), - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "sb-backfill".to_string(), + name: "backfill-sandbox".to_string(), + created_at_ms: 1_000_000, + labels: std::collections::HashMap::new(), + }), spec: Some(SandboxSpec { policy: None, ..Default::default() @@ -2241,6 +3334,440 @@ mod tests { assert_eq!(policy.process.unwrap().run_as_user, "sandbox"); } + async fn test_server_state() -> Arc { + let store = Arc::new( + Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(), + ); + let compute = new_test_runtime(store.clone()).await; + Arc::new(ServerState::new( + Config::new(None) + .with_database_url("sqlite::memory:?cache=shared") + .with_ssh_handshake_secret("test-secret"), + store, + compute, + SandboxIndex::new(), + SandboxWatchBus::new(), + TracingLogBus::new(), + Arc::new(SupervisorSessionRegistry::new()), + None, + )) + } + + #[tokio::test] + async fn draft_chunk_handler_lifecycle_round_trip() { + use openshell_core::proto::{ + GetDraftPolicyRequest, NetworkBinary, NetworkEndpoint, SandboxPhase, SandboxSpec, + }; + + let state = test_server_state().await; + let sandbox = Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "sb-draft-flow".to_string(), + name: "draft-flow".to_string(), + created_at_ms: 1_000_000, + labels: std::collections::HashMap::new(), + }), + spec: Some(SandboxSpec { + policy: None, + ..Default::default() + }), + phase: SandboxPhase::Ready as i32, + ..Default::default() + }; + state.store.put_message(&sandbox).await.unwrap(); + let sandbox_name = sandbox.object_name().to_string(); + + let proposed_rule = NetworkPolicyRule { + name: "allow_example".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.example.com".to_string(), + port: 443, + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }; + + let submit = handle_submit_policy_analysis( + &state, + Request::new(SubmitPolicyAnalysisRequest { + name: sandbox_name.clone(), + proposed_chunks: vec![PolicyChunk { + rule_name: "allow_example".to_string(), + proposed_rule: Some(proposed_rule.clone()), + rationale: "observed denied request".to_string(), + confidence: 0.85, + hit_count: 3, + first_seen_ms: 100, + last_seen_ms: 200, + binary: "/usr/bin/curl".to_string(), + ..Default::default() + }], + ..Default::default() + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(submit.accepted_chunks, 1); + assert_eq!(submit.rejected_chunks, 0); + + let draft_policy = handle_get_draft_policy( + &state, + Request::new(GetDraftPolicyRequest { + name: sandbox_name.clone(), + status_filter: String::new(), + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(draft_policy.draft_version, 1); + assert_eq!(draft_policy.chunks.len(), 1); + assert_eq!(draft_policy.chunks[0].status, "pending"); + let chunk_id = draft_policy.chunks[0].id.clone(); + + let approve = handle_approve_draft_chunk( + &state, + Request::new(ApproveDraftChunkRequest { + name: sandbox_name.clone(), + chunk_id: chunk_id.clone(), + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(approve.policy_version, 1); + assert!(!approve.policy_hash.is_empty()); + + let history_after_approve = handle_get_draft_history( + &state, + Request::new(GetDraftHistoryRequest { + name: sandbox_name.clone(), + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(history_after_approve.entries.len(), 2); + assert_eq!(history_after_approve.entries[0].event_type, "proposed"); + assert_eq!(history_after_approve.entries[1].event_type, "approved"); + assert_eq!(history_after_approve.entries[1].chunk_id, chunk_id); + + let policies_after_approve = handle_list_sandbox_policies( + &state, + Request::new(ListSandboxPoliciesRequest { + name: sandbox_name.clone(), + limit: 10, + offset: 0, + global: false, + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(policies_after_approve.revisions.len(), 1); + assert_eq!(policies_after_approve.revisions[0].version, 1); + + let undo = handle_undo_draft_chunk( + &state, + Request::new(UndoDraftChunkRequest { + name: sandbox_name.clone(), + chunk_id: chunk_id.clone(), + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(undo.policy_version, 2); + assert!(!undo.policy_hash.is_empty()); + + let draft_policy_after_undo = handle_get_draft_policy( + &state, + Request::new(GetDraftPolicyRequest { + name: sandbox_name.clone(), + status_filter: String::new(), + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(draft_policy_after_undo.chunks.len(), 1); + assert_eq!(draft_policy_after_undo.chunks[0].status, "pending"); + + let history_after_undo = handle_get_draft_history( + &state, + Request::new(GetDraftHistoryRequest { + name: sandbox_name.clone(), + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(history_after_undo.entries.len(), 1); + assert_eq!(history_after_undo.entries[0].event_type, "proposed"); + + let policies_after_undo = handle_list_sandbox_policies( + &state, + Request::new(ListSandboxPoliciesRequest { + name: sandbox_name.clone(), + limit: 10, + offset: 0, + global: false, + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(policies_after_undo.revisions.len(), 2); + assert_eq!(policies_after_undo.revisions[0].version, 2); + assert_eq!(policies_after_undo.revisions[1].version, 1); + + let cleared = handle_clear_draft_chunks( + &state, + Request::new(ClearDraftChunksRequest { + name: sandbox_name.clone(), + }), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(cleared.chunks_cleared, 1); + + let draft_policy_after_clear = handle_get_draft_policy( + &state, + Request::new(GetDraftPolicyRequest { + name: sandbox_name.clone(), + status_filter: String::new(), + }), + ) + .await + .unwrap() + .into_inner(); + assert!(draft_policy_after_clear.chunks.is_empty()); + + let history_after_clear = handle_get_draft_history( + &state, + Request::new(GetDraftHistoryRequest { name: sandbox_name }), + ) + .await + .unwrap() + .into_inner(); + assert!(history_after_clear.entries.is_empty()); + } + + #[tokio::test] + async fn draft_chunk_handlers_reject_cross_sandbox_chunk_ids() { + use openshell_core::proto::{NetworkBinary, NetworkEndpoint, SandboxPhase, SandboxSpec}; + + let state = test_server_state().await; + let sandbox_a = Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "sb-draft-owner".to_string(), + name: "draft-owner".to_string(), + created_at_ms: 1_000_000, + labels: std::collections::HashMap::new(), + }), + spec: Some(SandboxSpec { + policy: None, + ..Default::default() + }), + phase: SandboxPhase::Ready as i32, + ..Default::default() + }; + let sandbox_b = Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "sb-draft-other".to_string(), + name: "draft-other".to_string(), + created_at_ms: 1_000_001, + labels: std::collections::HashMap::new(), + }), + spec: Some(SandboxSpec { + policy: None, + ..Default::default() + }), + phase: SandboxPhase::Ready as i32, + ..Default::default() + }; + state.store.put_message(&sandbox_a).await.unwrap(); + state.store.put_message(&sandbox_b).await.unwrap(); + + let proposed_rule = NetworkPolicyRule { + name: "allow_example".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.example.com".to_string(), + port: 443, + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }; + + handle_submit_policy_analysis( + &state, + Request::new(SubmitPolicyAnalysisRequest { + name: sandbox_a.object_name().to_string(), + proposed_chunks: vec![PolicyChunk { + rule_name: "allow_example".to_string(), + proposed_rule: Some(proposed_rule.clone()), + rationale: "observed denied request".to_string(), + confidence: 0.85, + hit_count: 3, + first_seen_ms: 100, + last_seen_ms: 200, + binary: "/usr/bin/curl".to_string(), + ..Default::default() + }], + ..Default::default() + }), + ) + .await + .unwrap(); + + let draft_policy = handle_get_draft_policy( + &state, + Request::new(GetDraftPolicyRequest { + name: sandbox_a.object_name().to_string(), + status_filter: String::new(), + }), + ) + .await + .unwrap() + .into_inner(); + let chunk_id = draft_policy.chunks[0].id.clone(); + let other_name = sandbox_b.object_name().to_string(); + + let approve_err = handle_approve_draft_chunk( + &state, + Request::new(ApproveDraftChunkRequest { + name: other_name.clone(), + chunk_id: chunk_id.clone(), + }), + ) + .await + .unwrap_err(); + assert_eq!(approve_err.code(), Code::NotFound); + + let reject_err = handle_reject_draft_chunk( + &state, + Request::new(RejectDraftChunkRequest { + name: other_name.clone(), + chunk_id: chunk_id.clone(), + reason: "wrong sandbox".to_string(), + }), + ) + .await + .unwrap_err(); + assert_eq!(reject_err.code(), Code::NotFound); + + let edit_err = handle_edit_draft_chunk( + &state, + Request::new(EditDraftChunkRequest { + name: other_name.clone(), + chunk_id: chunk_id.clone(), + proposed_rule: Some(proposed_rule.clone()), + }), + ) + .await + .unwrap_err(); + assert_eq!(edit_err.code(), Code::NotFound); + + handle_approve_draft_chunk( + &state, + Request::new(ApproveDraftChunkRequest { + name: sandbox_a.object_name().to_string(), + chunk_id: chunk_id.clone(), + }), + ) + .await + .unwrap(); + + let undo_err = handle_undo_draft_chunk( + &state, + Request::new(UndoDraftChunkRequest { + name: other_name, + chunk_id, + }), + ) + .await + .unwrap_err(); + assert_eq!(undo_err.code(), Code::NotFound); + } + + #[test] + fn build_gateway_policy_audit_message_formats_ocsf_config_line() { + let message = build_gateway_policy_audit_message( + "sb-123", + "demo-sandbox", + "merged", + "gateway merged incremental policy op: add-allow api.github.com:443 [POST /repos/*/issues]", + 7, + "sha256:testhash", + ); + + assert_eq!( + message, + "CONFIG:MERGED [INFO] gateway merged incremental policy op: add-allow api.github.com:443 [POST /repos/*/issues] [version:v7 hash:sha256:testhash]" + ); + } + + #[test] + fn summarize_cli_policy_merge_op_formats_rest_allow_rules() { + let operation = PolicyMergeOp::AddAllowRules { + host: "api.github.com".to_string(), + port: 443, + rules: vec![L7Rule { + allow: Some(openshell_core::proto::L7Allow { + method: "POST".to_string(), + path: "/repos/*/issues".to_string(), + command: String::new(), + query: HashMap::new(), + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), + }), + }], + }; + + assert_eq!( + summarize_cli_policy_merge_op(&operation), + "add-allow api.github.com:443 [POST /repos/*/issues]" + ); + } + + #[test] + fn summarize_cli_policy_merge_op_formats_endpoint_additions() { + let operation = PolicyMergeOp::AddRule { + rule_name: "github_api".to_string(), + rule: NetworkPolicyRule { + name: "github_api".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.github.com".to_string(), + port: 443, + protocol: "rest".to_string(), + access: "read-only".to_string(), + enforcement: "enforce".to_string(), + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }, + }; + + assert_eq!( + summarize_cli_policy_merge_op(&operation), + "add-endpoint github_api endpoints=[api.github.com:443 protocol=rest access=read-only enforcement=enforce] binaries=[/usr/bin/curl]" + ); + } + // ---- merge_chunk_into_policy ---- #[tokio::test] @@ -2312,7 +3839,7 @@ mod tests { let sandbox_id = "sb-merge"; let initial_policy = SandboxPolicy { - network_policies: [( + network_policies: std::iter::once(( "test_server".to_string(), NetworkPolicyRule { name: "test_server".to_string(), @@ -2326,8 +3853,7 @@ mod tests { ..Default::default() }], }, - )] - .into_iter() + )) .collect(), ..Default::default() }; @@ -2412,7 +3938,7 @@ mod tests { let sandbox_id = "sb-new"; let initial_policy = SandboxPolicy { - network_policies: [( + network_policies: std::iter::once(( "existing_rule".to_string(), NetworkPolicyRule { name: "existing_rule".to_string(), @@ -2426,8 +3952,7 @@ mod tests { ..Default::default() }], }, - )] - .into_iter() + )) .collect(), ..Default::default() }; @@ -2488,6 +4013,91 @@ mod tests { assert!(policy.network_policies.contains_key("allow_10_0_0_5_8080")); } + #[tokio::test] + async fn concurrent_merge_batches_preserve_both_updates() { + use openshell_core::proto::{ + L7Allow, L7DenyRule, L7Rule, NetworkEndpoint, NetworkPolicyRule, SandboxPolicy, + }; + + let store = Store::connect("sqlite::memory:").await.unwrap(); + let sandbox_id = "sb-concurrent-merge"; + + let initial_policy = SandboxPolicy { + network_policies: std::iter::once(( + "github".to_string(), + NetworkPolicyRule { + name: "github".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.github.com".to_string(), + port: 443, + ports: vec![443], + protocol: "rest".to_string(), + access: "read-only".to_string(), + ..Default::default() + }], + ..Default::default() + }, + )) + .collect(), + ..Default::default() + }; + store + .put_policy_revision( + "p-seed", + sandbox_id, + 1, + &initial_policy.encode_to_vec(), + "seed-hash", + ) + .await + .unwrap(); + + let add_allow = [PolicyMergeOp::AddAllowRules { + host: "api.github.com".to_string(), + port: 443, + rules: vec![L7Rule { + allow: Some(L7Allow { + method: "POST".to_string(), + path: "/repos/*/issues".to_string(), + command: String::new(), + query: HashMap::new(), + operation_type: String::new(), + operation_name: String::new(), + fields: Vec::new(), + }), + }], + }]; + let add_deny = [PolicyMergeOp::AddDenyRules { + host: "api.github.com".to_string(), + port: 443, + deny_rules: vec![L7DenyRule { + method: "POST".to_string(), + path: "/admin".to_string(), + query: HashMap::new(), + ..Default::default() + }], + }]; + + let (left, right) = tokio::join!( + apply_merge_operations_with_retry(&store, sandbox_id, None, &add_allow), + apply_merge_operations_with_retry(&store, sandbox_id, None, &add_deny), + ); + + let mut versions = vec![left.unwrap().0, right.unwrap().0]; + versions.sort_unstable(); + assert_eq!(versions, vec![2, 3]); + + let latest = store.get_latest_policy(sandbox_id).await.unwrap().unwrap(); + assert_eq!(latest.version, 3); + + let policy = SandboxPolicy::decode(latest.policy_payload.as_slice()).unwrap(); + let endpoint = &policy.network_policies["github"].endpoints[0]; + assert!(endpoint.access.is_empty()); + assert_eq!(endpoint.rules.len(), 4); + assert_eq!(endpoint.deny_rules.len(), 1); + assert_eq!(endpoint.deny_rules[0].path, "/admin"); + } + // ---- validate_rule_not_always_blocked ---- #[test] @@ -2608,7 +4218,7 @@ mod tests { let global = StoredSettings::default(); let sandbox = StoredSettings::default(); let merged = merge_effective_settings(&global, &sandbox).unwrap(); - for registered in openshell_core::settings::REGISTERED_SETTINGS { + for registered in settings::REGISTERED_SETTINGS { let setting = merged .get(registered.key) .unwrap_or_else(|| panic!("missing registered key {}", registered.key)); @@ -2625,7 +4235,7 @@ mod tests { fn materialize_global_settings_includes_unset_registered_keys() { let global = StoredSettings::default(); let materialized = materialize_global_settings(&global).unwrap(); - for registered in openshell_core::settings::REGISTERED_SETTINGS { + for registered in settings::REGISTERED_SETTINGS { let setting = materialized .get(registered.key) .unwrap_or_else(|| panic!("missing registered key {}", registered.key)); @@ -2646,8 +4256,7 @@ mod tests { let encoded = hex::encode(policy.encode_to_vec()); let global = StoredSettings { revision: 1, - settings: [("policy".to_string(), StoredSettingValue::Bytes(encoded))] - .into_iter() + settings: std::iter::once(("policy".to_string(), StoredSettingValue::Bytes(encoded))) .collect(), }; @@ -2785,7 +4394,7 @@ mod tests { let global = StoredSettings::default(); let sandbox = StoredSettings::default(); let merged = merge_effective_settings(&global, &sandbox).unwrap(); - for registered in openshell_core::settings::REGISTERED_SETTINGS { + for registered in settings::REGISTERED_SETTINGS { let setting = merged.get(registered.key).unwrap(); assert_eq!(setting.scope, SettingScope::Unspecified as i32); assert!(setting.value.is_none()); @@ -2796,20 +4405,18 @@ mod tests { fn merge_effective_settings_policy_key_is_excluded() { let global = StoredSettings { revision: 1, - settings: [( + settings: std::iter::once(( "policy".to_string(), StoredSettingValue::Bytes("deadbeef".to_string()), - )] - .into_iter() + )) .collect(), }; let sandbox = StoredSettings { revision: 1, - settings: [( + settings: std::iter::once(( "policy".to_string(), StoredSettingValue::Bytes("cafebabe".to_string()), - )] - .into_iter() + )) .collect(), }; @@ -2818,25 +4425,9 @@ mod tests { } #[test] - fn sandbox_settings_id_has_prefix_preventing_collision() { - let sandbox_id = "abc-123"; - let settings_id = sandbox_settings_id(sandbox_id); - assert!(settings_id.starts_with("settings:")); - assert_ne!(settings_id, sandbox_id); - } - - #[test] - fn sandbox_settings_id_different_sandboxes_produce_different_ids() { - let id_a = sandbox_settings_id("sandbox-1"); - let id_b = sandbox_settings_id("sandbox-2"); - assert_ne!(id_a, id_b); - } - - #[test] - fn sandbox_settings_id_embeds_sandbox_id() { - let sandbox_id = "some-uuid-value"; - let settings_id = sandbox_settings_id(sandbox_id); - assert!(settings_id.contains(sandbox_id)); + fn sandbox_settings_names_match_sandbox_names() { + let sandbox_name = "my-sandbox"; + assert_eq!(sandbox_name, "my-sandbox"); } // ---- compute_config_revision ---- @@ -3046,17 +4637,17 @@ mod tests { .await .unwrap(); - let sandbox_id = "sb-uuid-123"; + let sandbox_name = "my-sandbox"; let mut settings = StoredSettings::default(); settings .settings .insert("dummy_int".to_string(), StoredSettingValue::Int(99)); settings.revision = 3; - save_sandbox_settings(&store, sandbox_id, "my-sandbox", &settings) + save_sandbox_settings(&store, sandbox_name, &settings) .await .unwrap(); - let loaded = load_sandbox_settings(&store, sandbox_id).await.unwrap(); + let loaded = load_sandbox_settings(&store, sandbox_name).await.unwrap(); assert_eq!(loaded.revision, 3); assert_eq!( loaded.settings.get("dummy_int"), @@ -3066,12 +4657,12 @@ mod tests { #[tokio::test] async fn concurrent_global_setting_mutations_are_serialized() { - let store = std::sync::Arc::new( + let store = Arc::new( Store::connect("sqlite::memory:?cache=shared") .await .unwrap(), ); - let mutex = std::sync::Arc::new(tokio::sync::Mutex::new(())); + let mutex = Arc::new(tokio::sync::Mutex::new(())); let n = 50; let mut handles = Vec::with_capacity(n); @@ -3101,7 +4692,7 @@ mod tests { #[tokio::test] async fn concurrent_global_setting_mutations_without_lock_can_lose_writes() { - let store = std::sync::Arc::new( + let store = Arc::new( Store::connect("sqlite::memory:?cache=shared") .await .unwrap(), @@ -3201,8 +4792,8 @@ mod tests { let loaded = load_global_settings(&store).await.unwrap(); assert!(!loaded.settings.contains_key("log_level")); - let sandbox_id = "test-sandbox-uuid"; - let mut sandbox_settings = load_sandbox_settings(&store, sandbox_id).await.unwrap(); + let sandbox_name = "test-sandbox"; + let mut sandbox_settings = load_sandbox_settings(&store, sandbox_name).await.unwrap(); let changed = upsert_setting_value( &mut sandbox_settings.settings, "log_level", @@ -3210,11 +4801,11 @@ mod tests { ); assert!(changed); sandbox_settings.revision = sandbox_settings.revision.wrapping_add(1); - save_sandbox_settings(&store, sandbox_id, "test-sandbox", &sandbox_settings) + save_sandbox_settings(&store, sandbox_name, &sandbox_settings) .await .unwrap(); - let reloaded = load_sandbox_settings(&store, sandbox_id).await.unwrap(); + let reloaded = load_sandbox_settings(&store, sandbox_name).await.unwrap(); assert_eq!( reloaded.settings.get("log_level"), Some(&StoredSettingValue::String("debug".to_string())), diff --git a/crates/openshell-server/src/grpc/provider.rs b/crates/openshell-server/src/grpc/provider.rs index bbf8f93ccf..3ff2547b15 100644 --- a/crates/openshell-server/src/grpc/provider.rs +++ b/crates/openshell-server/src/grpc/provider.rs @@ -5,7 +5,7 @@ #![allow(clippy::result_large_err)] // gRPC handlers return Result, Status> -use crate::persistence::{ObjectId, ObjectName, ObjectType, Store, generate_name}; +use crate::persistence::{ObjectType, Store, generate_name}; use openshell_core::proto::Provider; use prost::Message; use tonic::Status; @@ -33,9 +33,33 @@ pub(super) async fn create_provider_record( store: &Store, mut provider: Provider, ) -> Result { - if provider.name.is_empty() { - provider.name = generate_name(); + use crate::persistence::{ObjectName, current_time_ms}; + + // Initialize metadata if not present + if provider.metadata.is_none() { + let now_ms = current_time_ms() + .map_err(|e| Status::internal(format!("failed to get current time: {e}")))?; + provider.metadata = Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: uuid::Uuid::new_v4().to_string(), + name: generate_name(), + created_at_ms: now_ms, + labels: std::collections::HashMap::new(), + }); } + + // Auto-generate name if empty + if let Some(metadata) = provider.metadata.as_mut() { + if metadata.name.is_empty() { + metadata.name = generate_name(); + } + if metadata.id.is_empty() { + metadata.id = uuid::Uuid::new_v4().to_string(); + } + } + + // Ensure metadata is present and valid (must be non-None with non-empty id/name) + super::validation::validate_object_metadata(provider.metadata.as_ref(), "provider")?; + if provider.r#type.trim().is_empty() { return Err(Status::invalid_argument("provider.type is required")); } @@ -49,7 +73,7 @@ pub(super) async fn create_provider_record( validate_provider_fields(&provider)?; let existing = store - .get_message_by_name::(&provider.name) + .get_message_by_name::(provider.object_name()) .await .map_err(|e| Status::internal(format!("fetch provider failed: {e}")))?; @@ -57,8 +81,6 @@ pub(super) async fn create_provider_record( return Err(Status::already_exists("provider already exists")); } - provider.id = uuid::Uuid::new_v4().to_string(); - store .put_message(&provider) .await @@ -104,12 +126,14 @@ pub(super) async fn update_provider_record( store: &Store, provider: Provider, ) -> Result { - if provider.name.is_empty() { + use crate::persistence::ObjectName; + + if provider.object_name().is_empty() { return Err(Status::invalid_argument("provider.name is required")); } let existing = store - .get_message_by_name::(&provider.name) + .get_message_by_name::(provider.object_name()) .await .map_err(|e| Status::internal(format!("fetch provider failed: {e}")))?; @@ -127,13 +151,15 @@ pub(super) async fn update_provider_record( } let updated = Provider { - id: existing.id, - name: existing.name, + metadata: existing.metadata, r#type: existing.r#type, credentials: merge_map(existing.credentials, provider.credentials), config: merge_map(existing.config, provider.config), }; + // Ensure metadata is valid (defense in depth - existing.metadata should always be valid) + super::validation::validate_object_metadata(updated.metadata.as_ref(), "provider")?; + validate_provider_fields(&updated)?; store @@ -241,27 +267,18 @@ impl ObjectType for Provider { } } -impl ObjectId for Provider { - fn object_id(&self) -> &str { - &self.id - } -} - -impl ObjectName for Provider { - fn object_name(&self) -> &str { - &self.name - } -} - // --------------------------------------------------------------------------- // Handler wrappers called from the trait impl in mod.rs // --------------------------------------------------------------------------- use crate::ServerState; use openshell_core::proto::{ - CreateProviderRequest, DeleteProviderRequest, DeleteProviderResponse, GetProviderRequest, - ListProvidersRequest, ListProvidersResponse, ProviderResponse, UpdateProviderRequest, + CreateProviderRequest, DeleteProviderRequest, DeleteProviderResponse, + GetProviderProfileRequest, GetProviderRequest, ListProviderProfilesRequest, + ListProviderProfilesResponse, ListProvidersRequest, ListProvidersResponse, + ProviderProfileResponse, ProviderResponse, UpdateProviderRequest, }; +use openshell_providers::{default_profiles, get_default_profile}; use std::sync::Arc; use tonic::{Request, Response}; @@ -303,6 +320,40 @@ pub(super) async fn handle_list_providers( Ok(Response::new(ListProvidersResponse { providers })) } +pub(super) fn handle_list_provider_profiles( + _state: &Arc, + request: Request, +) -> Response { + let request = request.into_inner(); + let limit = clamp_limit(request.limit, 100, MAX_PAGE_SIZE) as usize; + let offset = request.offset as usize; + let profiles = default_profiles() + .iter() + .skip(offset) + .take(limit) + .map(openshell_providers::ProviderTypeProfile::to_proto) + .collect(); + + Response::new(ListProviderProfilesResponse { profiles }) +} + +pub(super) fn handle_get_provider_profile( + _state: &Arc, + request: Request, +) -> Result, Status> { + let id = request.into_inner().id; + if id.trim().is_empty() { + return Err(Status::invalid_argument("id is required")); + } + let profile = get_default_profile(id.trim()) + .ok_or_else(|| Status::not_found("provider profile not found"))? + .to_proto(); + + Ok(Response::new(ProviderProfileResponse { + profile: Some(profile), + })) +} + pub(super) async fn handle_update_provider( state: &Arc, request: Request, @@ -335,9 +386,21 @@ pub(super) async fn handle_delete_provider( #[cfg(test)] mod tests { use super::*; + use crate::ServerState; + use crate::compute::new_test_runtime; use crate::grpc::MAX_MAP_KEY_LEN; + use crate::sandbox_index::SandboxIndex; + use crate::sandbox_watch::SandboxWatchBus; + use crate::supervisor_session::SupervisorSessionRegistry; + use crate::tracing_bus::TracingLogBus; + use openshell_core::Config; + use openshell_core::proto::{ + GetProviderProfileRequest, ListProviderProfilesRequest, ProviderProfileCategory, + }; + use openshell_core::{ObjectId, ObjectName}; use std::collections::HashMap; - use tonic::Code; + use std::sync::Arc; + use tonic::{Code, Request}; #[test] fn env_key_validation_accepts_valid_keys() { @@ -358,8 +421,12 @@ mod tests { fn provider_with_values(name: &str, provider_type: &str) -> Provider { Provider { - id: String::new(), - name: name.to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: provider_type.to_string(), credentials: [ ("API_TOKEN".to_string(), "token-123".to_string()), @@ -376,6 +443,86 @@ mod tests { } } + async fn test_server_state() -> Arc { + let store = Arc::new( + Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(), + ); + let compute = new_test_runtime(store.clone()).await; + Arc::new(ServerState::new( + Config::new(None) + .with_database_url("sqlite::memory:?cache=shared") + .with_ssh_handshake_secret("test-secret"), + store, + compute, + SandboxIndex::new(), + SandboxWatchBus::new(), + TracingLogBus::new(), + Arc::new(SupervisorSessionRegistry::new()), + None, + )) + } + + #[tokio::test] + async fn list_provider_profiles_returns_built_in_profile_categories() { + let state = test_server_state().await; + let response = handle_list_provider_profiles( + &state, + Request::new(ListProviderProfilesRequest { + limit: 100, + offset: 0, + }), + ) + .into_inner(); + + let github = response + .profiles + .iter() + .find(|profile| profile.id == "github") + .expect("github profile should be listed"); + assert_eq!( + github.category, + ProviderProfileCategory::SourceControl as i32 + ); + assert!( + response + .profiles + .iter() + .all(|profile| profile.id != "generic"), + "generic remains a legacy provider type without a v2 profile" + ); + } + + #[tokio::test] + async fn get_provider_profile_returns_profile_or_not_found() { + let state = test_server_state().await; + let github = handle_get_provider_profile( + &state, + Request::new(GetProviderProfileRequest { + id: "github".to_string(), + }), + ) + .unwrap() + .into_inner() + .profile + .expect("github profile should be returned"); + assert_eq!(github.id, "github"); + assert_eq!( + github.category, + ProviderProfileCategory::SourceControl as i32 + ); + + let generic_err = handle_get_provider_profile( + &state, + Request::new(GetProviderProfileRequest { + id: "generic".to_string(), + }), + ) + .unwrap_err(); + assert_eq!(generic_err.code(), Code::NotFound); + } + #[tokio::test] async fn provider_crud_round_trip_and_semantics() { let store = Store::connect("sqlite::memory:?cache=shared") @@ -386,26 +533,30 @@ mod tests { let persisted = create_provider_record(&store, created.clone()) .await .unwrap(); - assert_eq!(persisted.name, "gitlab-local"); + assert_eq!(persisted.object_name(), "gitlab-local"); assert_eq!(persisted.r#type, "gitlab"); - assert!(!persisted.id.is_empty()); - let provider_id = persisted.id.clone(); + assert!(!persisted.object_id().is_empty()); + let provider_id = persisted.object_id().to_string(); let duplicate_err = create_provider_record(&store, created).await.unwrap_err(); assert_eq!(duplicate_err.code(), Code::AlreadyExists); let loaded = get_provider_record(&store, "gitlab-local").await.unwrap(); - assert_eq!(loaded.id, provider_id); + assert_eq!(loaded.object_id(), provider_id); let listed = list_provider_records(&store, 100, 0).await.unwrap(); assert_eq!(listed.len(), 1); - assert_eq!(listed[0].name, "gitlab-local"); + assert_eq!(listed[0].object_name(), "gitlab-local"); let updated = update_provider_record( &store, Provider { - id: String::new(), - name: "gitlab-local".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "gitlab-local".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), r#type: "gitlab".to_string(), credentials: std::iter::once(( "API_TOKEN".to_string(), @@ -418,7 +569,7 @@ mod tests { ) .await .unwrap(); - assert_eq!(updated.id, provider_id); + assert_eq!(updated.object_id(), provider_id); assert_eq!(updated.credentials.len(), 2); assert_eq!( updated.credentials.get("API_TOKEN"), @@ -473,8 +624,12 @@ mod tests { let create_missing_type = create_provider_record( &store, Provider { - id: String::new(), - name: "bad-provider".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "bad-provider".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), r#type: String::new(), credentials: HashMap::new(), config: HashMap::new(), @@ -493,8 +648,12 @@ mod tests { let update_missing_err = update_provider_record( &store, Provider { - id: String::new(), - name: "missing".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "missing".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), r#type: String::new(), credentials: HashMap::new(), config: HashMap::new(), @@ -517,8 +676,12 @@ mod tests { let updated = update_provider_record( &store, Provider { - id: String::new(), - name: "noop-test".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "noop-test".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), r#type: String::new(), credentials: HashMap::new(), config: HashMap::new(), @@ -527,7 +690,7 @@ mod tests { .await .unwrap(); - assert_eq!(updated.id, persisted.id); + assert_eq!(updated.object_id(), persisted.object_id()); assert_eq!(updated.r#type, "nvidia"); assert_eq!(updated.credentials.len(), 2); assert_eq!( @@ -560,8 +723,12 @@ mod tests { let updated = update_provider_record( &store, Provider { - id: String::new(), - name: "delete-key-test".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "delete-key-test".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: String::new(), credentials: std::iter::once(("SECONDARY".to_string(), String::new())).collect(), config: std::iter::once(("region".to_string(), String::new())).collect(), @@ -575,13 +742,13 @@ mod tests { updated.credentials.get("API_TOKEN"), Some(&"REDACTED".to_string()) ); - assert!(updated.credentials.get("SECONDARY").is_none()); + assert!(!updated.credentials.contains_key("SECONDARY")); assert_eq!(updated.config.len(), 1); assert_eq!( updated.config.get("endpoint"), Some(&"https://example.com".to_string()) ); - assert!(updated.config.get("region").is_none()); + assert!(!updated.config.contains_key("region")); let stored: Provider = store .get_message_by_name("delete-key-test") .await @@ -592,7 +759,7 @@ mod tests { stored.credentials.get("API_TOKEN"), Some(&"token-123".to_string()) ); - assert!(stored.credentials.get("SECONDARY").is_none()); + assert!(!stored.credentials.contains_key("SECONDARY")); } #[tokio::test] @@ -607,8 +774,12 @@ mod tests { let updated = update_provider_record( &store, Provider { - id: String::new(), - name: "type-preserve-test".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "type-preserve-test".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: String::new(), credentials: HashMap::new(), config: HashMap::new(), @@ -632,8 +803,12 @@ mod tests { let err = update_provider_record( &store, Provider { - id: String::new(), - name: "type-change-test".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "type-change-test".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: "openai".to_string(), credentials: HashMap::new(), config: HashMap::new(), @@ -659,8 +834,12 @@ mod tests { let err = update_provider_record( &store, Provider { - id: String::new(), - name: "validate-merge-test".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "validate-merge-test".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: String::new(), credentials: std::iter::once((oversized_key, "value".to_string())).collect(), config: HashMap::new(), @@ -683,8 +862,12 @@ mod tests { async fn resolve_provider_env_injects_credentials() { let store = Store::connect("sqlite::memory:").await.unwrap(); let provider = Provider { - id: String::new(), - name: "claude-local".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "claude-local".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: "claude".to_string(), credentials: [ ("ANTHROPIC_API_KEY".to_string(), "sk-abc".to_string()), @@ -722,8 +905,12 @@ mod tests { async fn resolve_provider_env_skips_invalid_credential_keys() { let store = Store::connect("sqlite::memory:").await.unwrap(); let provider = Provider { - id: String::new(), - name: "test-provider".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "test-provider".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: "test".to_string(), credentials: [ ("VALID_KEY".to_string(), "value".to_string()), @@ -750,8 +937,12 @@ mod tests { create_provider_record( &store, Provider { - id: String::new(), - name: "claude-local".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "claude-local".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: "claude".to_string(), credentials: std::iter::once(( "ANTHROPIC_API_KEY".to_string(), @@ -766,8 +957,12 @@ mod tests { create_provider_record( &store, Provider { - id: String::new(), - name: "gitlab-local".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "gitlab-local".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: "gitlab".to_string(), credentials: std::iter::once(("GITLAB_TOKEN".to_string(), "glpat-xyz".to_string())) .collect(), @@ -793,8 +988,12 @@ mod tests { create_provider_record( &store, Provider { - id: String::new(), - name: "provider-a".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "provider-a".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: "claude".to_string(), credentials: std::iter::once(("SHARED_KEY".to_string(), "first-value".to_string())) .collect(), @@ -806,8 +1005,12 @@ mod tests { create_provider_record( &store, Provider { - id: String::new(), - name: "provider-b".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "provider-b".to_string(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: "gitlab".to_string(), credentials: std::iter::once(( "SHARED_KEY".to_string(), @@ -838,8 +1041,12 @@ mod tests { create_provider_record( &store, Provider { - id: String::new(), - name: "my-claude".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: "my-claude".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), r#type: "claude".to_string(), credentials: std::iter::once(( "ANTHROPIC_API_KEY".to_string(), @@ -853,9 +1060,12 @@ mod tests { .unwrap(); let sandbox = Sandbox { - id: "sandbox-001".to_string(), - name: "test-sandbox".to_string(), - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "sandbox-001".to_string(), + name: "test-sandbox".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), spec: Some(SandboxSpec { providers: vec!["my-claude".to_string()], ..SandboxSpec::default() @@ -886,9 +1096,12 @@ mod tests { let store = Store::connect("sqlite::memory:").await.unwrap(); let sandbox = Sandbox { - id: "sandbox-002".to_string(), - name: "empty-sandbox".to_string(), - namespace: "default".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "sandbox-002".to_string(), + name: "empty-sandbox".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), spec: Some(SandboxSpec::default()), status: None, phase: SandboxPhase::Ready as i32, diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index 8e59308265..ed1b4cdfc4 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -22,13 +22,11 @@ use openshell_core::proto::{ use openshell_core::proto::{Sandbox, SandboxPhase, SandboxTemplate, SshSession}; use prost::Message; use std::sync::Arc; -use tokio::io::AsyncReadExt; -use tokio::io::AsyncWriteExt; use tokio::net::{TcpListener, TcpStream}; use tokio::sync::mpsc; use tokio_stream::wrappers::ReceiverStream; use tonic::{Request, Response, Status}; -use tracing::{debug, info, warn}; +use tracing::{info, warn}; use russh::ChannelMsg; use russh::client::AuthResult; @@ -48,6 +46,8 @@ pub(super) async fn handle_create_sandbox( state: &Arc, request: Request, ) -> Result, Status> { + use crate::persistence::current_time_ms; + let request = request.into_inner(); let spec = request .spec @@ -56,6 +56,12 @@ pub(super) async fn handle_create_sandbox( // Validate field sizes before any I/O (fail fast on oversized payloads). validate_sandbox_spec(&request.name, &spec)?; + // Validate labels (keys and values must meet Kubernetes requirements). + for (key, value) in &request.labels { + crate::grpc::validation::validate_label_key(key)?; + crate::grpc::validation::validate_label_value(value)?; + } + // Validate provider names exist (fail fast). for name in &spec.providers { state @@ -86,18 +92,26 @@ pub(super) async fn handle_create_sandbox( } else { request.name.clone() }; - let namespace = state.config.sandbox_namespace.clone(); + + let now_ms = current_time_ms() + .map_err(|e| Status::internal(format!("failed to get current time: {e}")))?; let sandbox = Sandbox { - id: id.clone(), - name: name.clone(), - namespace, + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: id.clone(), + name: name.clone(), + created_at_ms: now_ms, + labels: request.labels.clone(), + }), spec: Some(spec), status: None, phase: SandboxPhase::Provisioning as i32, - ..Default::default() + current_policy_version: 0, }; + // Ensure metadata is valid (defense in depth - should always be true for server-constructed metadata) + super::validation::validate_object_metadata(sandbox.metadata.as_ref(), "sandbox")?; + state .compute .validate_sandbox_create(&sandbox) @@ -110,8 +124,8 @@ pub(super) async fn handle_create_sandbox( let sandbox = state.compute.create_sandbox(sandbox).await?; info!( - sandbox_id = %sandbox.id, - sandbox_name = %sandbox.name, + sandbox_id = %id, + sandbox_name = %name, "CreateSandbox request completed successfully" ); Ok(Response::new(SandboxResponse { @@ -146,17 +160,32 @@ pub(super) async fn handle_list_sandboxes( ) -> Result, Status> { let request = request.into_inner(); let limit = clamp_limit(request.limit, 100, MAX_PAGE_SIZE); - let records = state - .store - .list(Sandbox::object_type(), limit, request.offset) - .await - .map_err(|e| Status::internal(format!("list sandboxes failed: {e}")))?; + + // If no label selector is provided, use the unfiltered list path + let records = if request.label_selector.is_empty() { + state + .store + .list(Sandbox::object_type(), limit, request.offset) + .await + .map_err(|e| Status::internal(format!("list sandboxes failed: {e}")))? + } else { + crate::grpc::validation::validate_label_selector(&request.label_selector)?; + state + .store + .list_with_selector( + Sandbox::object_type(), + &request.label_selector, + limit, + request.offset, + ) + .await + .map_err(|e| Status::internal(format!("list sandboxes with selector failed: {e}")))? + }; let mut sandboxes = Vec::with_capacity(records.len()); for record in records { - let mut sandbox = Sandbox::decode(record.payload.as_slice()) + let sandbox = Sandbox::decode(record.payload.as_slice()) .map_err(|e| Status::internal(format!("decode sandbox failed: {e}")))?; - sandbox.created_at_ms = record.created_at_ms; sandboxes.push(sandbox); } @@ -413,6 +442,8 @@ pub(super) async fn handle_exec_sandbox( state: &Arc, request: Request, ) -> Result>>, Status> { + use openshell_core::ObjectId; + let req = request.into_inner(); if req.sandbox_id.is_empty() { return Err(Status::invalid_argument("sandbox_id is required")); @@ -438,27 +469,56 @@ pub(super) async fn handle_exec_sandbox( return Err(Status::failed_precondition("sandbox is not ready")); } - let (target_host, target_port) = resolve_sandbox_exec_target(state, &sandbox).await?; + // Open a relay channel through the supervisor session. Use a 15s + // session-wait timeout — enough to cover a transient supervisor + // reconnect, but shorter than `/connect/ssh` since `ExecSandbox` is + // typically called during normal operation (not right after create). + let (channel_id, relay_rx) = state + .supervisor_sessions + .open_relay(sandbox.object_id(), std::time::Duration::from_secs(15)) + .await + .map_err(|e| Status::unavailable(format!("supervisor relay failed: {e}")))?; + let command_str = build_remote_exec_command(&req) .map_err(|e| Status::invalid_argument(format!("command construction failed: {e}")))?; let stdin_payload = req.stdin; let timeout_seconds = req.timeout_seconds; let request_tty = req.tty; - let sandbox_id = sandbox.id; - let handshake_secret = state.config.ssh_handshake_secret.clone(); + + let sandbox_id = sandbox.object_id().to_string(); let (tx, rx) = mpsc::channel::>(256); tokio::spawn(async move { - if let Err(err) = stream_exec_over_ssh( + // Wait for the supervisor's reverse CONNECT to deliver the relay stream. + let relay_stream = match tokio::time::timeout(std::time::Duration::from_secs(10), relay_rx) + .await + { + Ok(Ok(stream)) => stream, + Ok(Err(_)) => { + warn!(sandbox_id = %sandbox_id, channel_id = %channel_id, "ExecSandbox: relay channel dropped"); + let _ = tx + .send(Err(Status::unavailable("relay channel dropped"))) + .await; + return; + } + Err(_) => { + warn!(sandbox_id = %sandbox_id, channel_id = %channel_id, "ExecSandbox: relay open timed out"); + let _ = tx + .send(Err(Status::deadline_exceeded("relay open timed out"))) + .await; + return; + } + }; + + if let Err(err) = stream_exec_over_relay( tx.clone(), &sandbox_id, - &target_host, - target_port, + &channel_id, + relay_stream, &command_str, stdin_payload, timeout_seconds, request_tty, - &handshake_secret, ) .await { @@ -503,15 +563,21 @@ pub(super) async fn handle_create_ssh_session( 0 }; let session = SshSession { - id: token.clone(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: token.clone(), + name: generate_name(), + created_at_ms: now_ms, + labels: std::collections::HashMap::new(), + }), sandbox_id: req.sandbox_id.clone(), token: token.clone(), - created_at_ms: now_ms, revoked: false, - name: generate_name(), expires_at_ms, }; + // Ensure metadata is valid (defense in depth - should always be true for server-constructed metadata) + super::validation::validate_object_metadata(session.metadata.as_ref(), "ssh_session")?; + state .store .put_message(&session) @@ -584,16 +650,6 @@ fn resolve_gateway(config: &openshell_core::Config) -> (String, u16) { (host, port) } -async fn resolve_sandbox_exec_target( - state: &ServerState, - sandbox: &Sandbox, -) -> Result<(String, u16), Status> { - match state.compute.resolve_sandbox_endpoint(sandbox).await? { - crate::compute::ResolvedEndpoint::Ip(ip, port) => Ok((ip.to_string(), port)), - crate::compute::ResolvedEndpoint::Host(host, port) => Ok((host, port)), - } -} - /// Shell-escape a value for embedding in a POSIX shell command. /// /// Wraps unsafe values in single quotes with the standard `'\''` idiom for @@ -625,7 +681,7 @@ const MAX_COMMAND_STRING_LEN: usize = 256 * 1024; // 256 KiB fn build_remote_exec_command(req: &ExecSandboxRequest) -> Result { let mut parts = Vec::new(); let mut env_entries = req.environment.iter().collect::>(); - env_entries.sort_by(|(a, _), (b, _)| a.cmp(b)); + env_entries.sort_by_key(|(a, _)| *a); for (key, value) in env_entries { parts.push(format!("{key}={}", shell_escape(value)?)); } @@ -646,133 +702,72 @@ fn build_remote_exec_command(req: &ExecSandboxRequest) -> Result Ok(result) } -/// Maximum number of attempts when establishing the SSH transport to a sandbox. -const SSH_CONNECT_MAX_ATTEMPTS: u32 = 6; - -/// Initial backoff duration between SSH connection retries. -const SSH_CONNECT_INITIAL_BACKOFF: std::time::Duration = std::time::Duration::from_millis(250); - -/// Maximum backoff duration between SSH connection retries. -const SSH_CONNECT_MAX_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2); - -/// Returns `true` if the gRPC status represents a transient SSH connection error. -fn is_retryable_ssh_error(status: &Status) -> bool { - if status.code() != tonic::Code::Internal { - return false; - } - let msg = status.message(); - msg.contains("Connection reset by peer") - || msg.contains("Connection refused") - || msg.contains("failed to establish ssh transport") - || msg.contains("failed to connect to ssh proxy") - || msg.contains("failed to start ssh proxy") -} - +/// Execute a command over an SSH transport relayed through a supervisor session. +/// +/// This is the relay equivalent of `stream_exec_over_ssh`. Instead of dialing a +/// sandbox endpoint directly, the SSH transport runs over a `DuplexStream` that +/// is bridged to the supervisor's local SSH daemon via a reverse HTTP CONNECT +/// tunnel. #[allow(clippy::too_many_arguments)] -async fn stream_exec_over_ssh( +async fn stream_exec_over_relay( tx: mpsc::Sender>, sandbox_id: &str, - target_host: &str, - target_port: u16, + channel_id: &str, + relay_stream: tokio::io::DuplexStream, command: &str, stdin_payload: Vec, timeout_seconds: u32, request_tty: bool, - handshake_secret: &str, ) -> Result<(), Status> { let command_preview: String = command.chars().take(120).collect(); info!( sandbox_id = %sandbox_id, - target_host = %target_host, - target_port, + channel_id = %channel_id, command_len = command.len(), stdin_len = stdin_payload.len(), command_preview = %command_preview, - "ExecSandbox command started" + "ExecSandbox (relay): command started" ); - let (exit_code, proxy_task) = { - let mut last_err: Option = None; - - let mut result = None; - for attempt in 0..SSH_CONNECT_MAX_ATTEMPTS { - if attempt > 0 { - let backoff = (SSH_CONNECT_INITIAL_BACKOFF * 2u32.pow(attempt - 1)) - .min(SSH_CONNECT_MAX_BACKOFF); - warn!( - sandbox_id = %sandbox_id, - attempt = attempt + 1, - backoff_ms = %backoff.as_millis(), - error = %last_err.as_ref().unwrap(), - "Retrying SSH transport establishment" - ); - tokio::time::sleep(backoff).await; - } + let (local_proxy_port, proxy_task) = start_single_use_ssh_proxy_over_relay(relay_stream) + .await + .map_err(|e| Status::internal(format!("failed to start relay proxy: {e}")))?; + + let exec = run_exec_with_russh( + local_proxy_port, + command, + stdin_payload, + request_tty, + tx.clone(), + ); - let (local_proxy_port, proxy_task) = match start_single_use_ssh_proxy( - target_host, - target_port, - handshake_secret, - ) - .await - { - Ok(v) => v, - Err(e) => { - last_err = Some(Status::internal(format!("failed to start ssh proxy: {e}"))); - continue; - } - }; - - let exec = run_exec_with_russh( - local_proxy_port, - command, - stdin_payload.clone(), - request_tty, - tx.clone(), - ); + let exec_result = if timeout_seconds == 0 { + exec.await + } else if let Ok(r) = tokio::time::timeout( + std::time::Duration::from_secs(u64::from(timeout_seconds)), + exec, + ) + .await + { + r + } else { + let _ = tx + .send(Ok(ExecSandboxEvent { + payload: Some(openshell_core::proto::exec_sandbox_event::Payload::Exit( + ExecSandboxExit { exit_code: 124 }, + )), + })) + .await; + let _ = proxy_task.await; + return Ok(()); + }; - let exec_result = if timeout_seconds == 0 { - exec.await - } else if let Ok(r) = tokio::time::timeout( - std::time::Duration::from_secs(u64::from(timeout_seconds)), - exec, - ) - .await - { - r - } else { - let _ = tx - .send(Ok(ExecSandboxEvent { - payload: Some(openshell_core::proto::exec_sandbox_event::Payload::Exit( - ExecSandboxExit { exit_code: 124 }, - )), - })) - .await; - let _ = proxy_task.await; - return Ok(()); - }; - - match exec_result { - Ok(exit_code) => { - result = Some((exit_code, proxy_task)); - break; - } - Err(status) => { - let _ = proxy_task.await; - if is_retryable_ssh_error(&status) && attempt + 1 < SSH_CONNECT_MAX_ATTEMPTS { - last_err = Some(status); - continue; - } - return Err(status); - } - } + let exit_code = match exec_result { + Ok(code) => code, + Err(status) => { + let _ = proxy_task.await; + return Err(status); } - - result.ok_or_else(|| { - last_err.unwrap_or_else(|| { - Status::internal("ssh connection failed after exhausting retries") - }) - })? }; let _ = proxy_task.await; @@ -788,6 +783,28 @@ async fn stream_exec_over_ssh( Ok(()) } +/// Create a localhost SSH proxy that bridges to a relay `DuplexStream`. +/// +/// The proxy forwards raw SSH bytes between the `russh` client and the relay. +/// The supervisor bridges the relay to its Unix-socket SSH daemon; filesystem +/// permissions on that socket are the only access-control boundary. +async fn start_single_use_ssh_proxy_over_relay( + mut relay_stream: tokio::io::DuplexStream, +) -> Result<(u16, tokio::task::JoinHandle<()>), Box> { + let listener = TcpListener::bind(("127.0.0.1", 0)).await?; + let port = listener.local_addr()?.port(); + + let task = tokio::spawn(async move { + let Ok((mut client_conn, _)) = listener.accept().await else { + warn!("SSH relay proxy: failed to accept local connection"); + return; + }; + let _ = tokio::io::copy_bidirectional(&mut client_conn, &mut relay_stream).await; + }); + + Ok((port, task)) +} + #[derive(Debug, Clone, Copy)] struct SandboxSshClientHandler; @@ -914,148 +931,6 @@ async fn run_exec_with_russh( Ok(exit_code.unwrap_or(1)) } -/// Check whether an IP address is safe to use as an SSH proxy target. -fn is_safe_ssh_proxy_target(ip: std::net::IpAddr) -> bool { - match ip { - std::net::IpAddr::V4(v4) => !v4.is_loopback() && !v4.is_link_local(), - std::net::IpAddr::V6(v6) => { - if v6.is_loopback() { - return false; - } - if let Some(v4) = v6.to_ipv4_mapped() { - return !v4.is_loopback() && !v4.is_link_local(); - } - true - } - } -} - -async fn start_single_use_ssh_proxy( - target_host: &str, - target_port: u16, - handshake_secret: &str, -) -> Result<(u16, tokio::task::JoinHandle<()>), Box> { - let listener = TcpListener::bind(("127.0.0.1", 0)).await?; - let port = listener.local_addr()?.port(); - let target_host = target_host.to_string(); - let handshake_secret = handshake_secret.to_string(); - - let task = tokio::spawn(async move { - let Ok((mut client_conn, _)) = listener.accept().await else { - warn!("SSH proxy: failed to accept local connection"); - return; - }; - - let addr_str = format!("{target_host}:{target_port}"); - let resolved = match tokio::net::lookup_host(&addr_str).await { - Ok(mut addrs) => { - if let Some(addr) = addrs.next() { - addr - } else { - warn!(target_host = %target_host, "SSH proxy: DNS resolution returned no addresses"); - return; - } - } - Err(e) => { - warn!(target_host = %target_host, error = %e, "SSH proxy: DNS resolution failed"); - return; - } - }; - - if !is_safe_ssh_proxy_target(resolved.ip()) { - warn!( - target_host = %target_host, - resolved_ip = %resolved.ip(), - "SSH proxy: target resolved to blocked IP range (loopback or link-local)" - ); - return; - } - - debug!( - target_host = %target_host, - resolved_ip = %resolved.ip(), - target_port, - "SSH proxy: connecting to validated target" - ); - - let Ok(mut sandbox_conn) = TcpStream::connect(resolved).await else { - warn!(target_host = %target_host, resolved_ip = %resolved.ip(), target_port, "SSH proxy: failed to connect to sandbox"); - return; - }; - let Ok(preface) = build_preface(&uuid::Uuid::new_v4().to_string(), &handshake_secret) - else { - warn!("SSH proxy: failed to build handshake preface"); - return; - }; - if let Err(e) = sandbox_conn.write_all(preface.as_bytes()).await { - warn!(error = %e, "SSH proxy: failed to send handshake preface"); - return; - } - let mut response = String::new(); - if let Err(e) = read_line(&mut sandbox_conn, &mut response).await { - warn!(error = %e, "SSH proxy: failed to read handshake response"); - return; - } - if response.trim() != "OK" { - warn!(response = %response.trim(), "SSH proxy: handshake rejected by sandbox"); - return; - } - let _ = tokio::io::copy_bidirectional(&mut client_conn, &mut sandbox_conn).await; - }); - - Ok((port, task)) -} - -fn build_preface( - token: &str, - secret: &str, -) -> Result> { - let timestamp = i64::try_from( - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map_err(|_| "time error")? - .as_secs(), - ) - .map_err(|_| "time error")?; - let nonce = uuid::Uuid::new_v4().to_string(); - let payload = format!("{token}|{timestamp}|{nonce}"); - let signature = hmac_sha256(secret.as_bytes(), payload.as_bytes()); - Ok(format!("NSSH1 {token} {timestamp} {nonce} {signature}\n")) -} - -async fn read_line( - stream: &mut TcpStream, - buf: &mut String, -) -> Result<(), Box> { - let mut bytes = Vec::new(); - loop { - let mut byte = [0_u8; 1]; - let n = stream.read(&mut byte).await?; - if n == 0 { - break; - } - if byte[0] == b'\n' { - break; - } - bytes.push(byte[0]); - if bytes.len() > 1024 { - break; - } - } - *buf = String::from_utf8_lossy(&bytes).to_string(); - Ok(()) -} - -fn hmac_sha256(key: &[u8], data: &[u8]) -> String { - use hmac::{Hmac, Mac}; - use sha2::Sha256; - - let mut mac = Hmac::::new_from_slice(key).expect("hmac key"); - mac.update(data); - let result = mac.finalize().into_bytes(); - hex::encode(result) -} - // --------------------------------------------------------------------------- // Tests // --------------------------------------------------------------------------- @@ -1126,9 +1001,7 @@ mod tests { "-c".to_string(), "print('ok')".to_string(), ], - environment: [("HOME".to_string(), "/home/user".to_string())] - .into_iter() - .collect(), + environment: std::iter::once(("HOME".to_string(), "/home/user".to_string())).collect(), workdir: "/workspace".to_string(), ..Default::default() }; @@ -1161,59 +1034,6 @@ mod tests { assert!(build_remote_exec_command(&req).is_err()); } - // ---- is_safe_ssh_proxy_target ---- - - #[test] - fn ssh_proxy_target_allows_pod_network_ips() { - use std::net::{IpAddr, Ipv4Addr}; - assert!(is_safe_ssh_proxy_target(IpAddr::V4(Ipv4Addr::new( - 10, 0, 0, 5 - )))); - assert!(is_safe_ssh_proxy_target(IpAddr::V4(Ipv4Addr::new( - 172, 16, 0, 1 - )))); - assert!(is_safe_ssh_proxy_target(IpAddr::V4(Ipv4Addr::new( - 192, 168, 1, 100 - )))); - } - - #[test] - fn ssh_proxy_target_blocks_loopback() { - use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; - assert!(!is_safe_ssh_proxy_target(IpAddr::V4(Ipv4Addr::new( - 127, 0, 0, 1 - )))); - assert!(!is_safe_ssh_proxy_target(IpAddr::V4(Ipv4Addr::new( - 127, 0, 0, 2 - )))); - assert!(!is_safe_ssh_proxy_target(IpAddr::V6(Ipv6Addr::LOCALHOST))); - } - - #[test] - fn ssh_proxy_target_blocks_link_local() { - use std::net::{IpAddr, Ipv4Addr}; - assert!(!is_safe_ssh_proxy_target(IpAddr::V4(Ipv4Addr::new( - 169, 254, 169, 254 - )))); - assert!(!is_safe_ssh_proxy_target(IpAddr::V4(Ipv4Addr::new( - 169, 254, 0, 1 - )))); - } - - #[test] - fn ssh_proxy_target_blocks_ipv4_mapped_ipv6_loopback() { - use std::net::IpAddr; - let ip: IpAddr = "::ffff:127.0.0.1".parse().unwrap(); - assert!(!is_safe_ssh_proxy_target(ip)); - } - - #[test] - fn ssh_proxy_target_blocks_ipv4_mapped_ipv6_link_local() { - use std::net::IpAddr; - let ip: IpAddr = "::ffff:169.254.169.254".parse().unwrap(); - assert!(!is_safe_ssh_proxy_target(ip)); - } - // ---- petname / generate_name ---- #[test] diff --git a/crates/openshell-server/src/grpc/validation.rs b/crates/openshell-server/src/grpc/validation.rs index 8d53da6a0a..160b7e0312 100644 --- a/crates/openshell-server/src/grpc/validation.rs +++ b/crates/openshell-server/src/grpc/validation.rs @@ -241,10 +241,10 @@ pub(super) fn validate_string_map( /// Validate field sizes on a `Provider` before persisting. pub(super) fn validate_provider_fields(provider: &Provider) -> Result<(), Status> { - if provider.name.len() > MAX_NAME_LEN { + let name_len = provider.metadata.as_ref().map_or(0, |m| m.name.len()); + if name_len > MAX_NAME_LEN { return Err(Status::invalid_argument(format!( - "provider.name exceeds maximum length ({} > {MAX_NAME_LEN})", - provider.name.len() + "provider.name exceeds maximum length ({name_len} > {MAX_NAME_LEN})" ))); } if provider.r#type.len() > MAX_PROVIDER_TYPE_LEN { @@ -270,6 +270,252 @@ pub(super) fn validate_provider_fields(provider: &Provider) -> Result<(), Status Ok(()) } +// --------------------------------------------------------------------------- +// Label selector validation +// --------------------------------------------------------------------------- + +/// Validate a label selector string format. +/// +/// Format: "key1=value1,key2=value2" +/// Returns `INVALID_ARGUMENT` if the selector has invalid format. +/// Validate a label key according to Kubernetes requirements. +/// +/// Label keys have an optional prefix and required name, separated by `/`: +/// - Prefix (optional): DNS subdomain format, max 253 chars +/// - Name (required): alphanumeric + `-._`, max 63 chars, must start/end with alphanumeric +/// - Total length including `/` must not exceed 253 chars +/// +/// Examples: `app`, `kubernetes.io/app`, `example.com/my-label` +/// +/// See: +pub(super) fn validate_label_key(key: &str) -> Result<(), Status> { + if key.is_empty() { + return Err(Status::invalid_argument("label key cannot be empty")); + } + + if key.len() > 253 { + return Err(Status::invalid_argument(format!( + "label key exceeds 253 characters: '{key}'" + ))); + } + + // Split into optional prefix and required name + let (prefix, name) = if let Some((p, n)) = key.split_once('/') { + (Some(p), n) + } else { + (None, key) + }; + + // Validate name segment (required, max 63 chars) + if name.is_empty() { + return Err(Status::invalid_argument(format!( + "label key name segment cannot be empty: '{key}'" + ))); + } + + if name.len() > 63 { + return Err(Status::invalid_argument(format!( + "label key name segment exceeds 63 characters: '{key}'" + ))); + } + + // Name must contain only alphanumeric, hyphens, underscores, and dots + if !name + .chars() + .all(|c| c.is_alphanumeric() || c == '-' || c == '_' || c == '.') + { + return Err(Status::invalid_argument(format!( + "label key name segment contains invalid characters (must be alphanumeric, '-', '_', or '.'): '{key}'" + ))); + } + + // Name must start and end with alphanumeric + let first = name.chars().next().unwrap(); // safe: we checked !is_empty() + let last = name.chars().last().unwrap(); + if !first.is_alphanumeric() { + return Err(Status::invalid_argument(format!( + "label key name segment must start with alphanumeric character: '{key}'" + ))); + } + if !last.is_alphanumeric() { + return Err(Status::invalid_argument(format!( + "label key name segment must end with alphanumeric character: '{key}'" + ))); + } + + // Validate prefix if present (DNS subdomain format) + if let Some(prefix) = prefix { + if prefix.is_empty() { + return Err(Status::invalid_argument(format!( + "label key prefix cannot be empty when '/' is present: '{key}'" + ))); + } + + if prefix.len() > 253 { + return Err(Status::invalid_argument(format!( + "label key prefix exceeds 253 characters: '{key}'" + ))); + } + + // DNS subdomain: lowercase alphanumeric, hyphens, and dots only + if !prefix + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-' || c == '.') + { + return Err(Status::invalid_argument(format!( + "label key prefix must be a DNS subdomain (lowercase alphanumeric, '-', '.'): '{key}'" + ))); + } + + // Must not start or end with hyphen or dot + if prefix.starts_with('-') + || prefix.starts_with('.') + || prefix.ends_with('-') + || prefix.ends_with('.') + { + return Err(Status::invalid_argument(format!( + "label key prefix cannot start or end with '-' or '.': '{key}'" + ))); + } + + // Must not contain consecutive dots + if prefix.contains("..") { + return Err(Status::invalid_argument(format!( + "label key prefix cannot contain consecutive dots: '{key}'" + ))); + } + } + + Ok(()) +} + +/// Validate a label value according to Kubernetes requirements. +/// +/// Label values: +/// - Can be empty (Kubernetes allows empty values) +/// - Max 63 characters +/// - If non-empty, must contain only alphanumeric, hyphens, underscores, and dots +/// - If non-empty, must start and end with alphanumeric character +/// +/// See: +pub(super) fn validate_label_value(value: &str) -> Result<(), Status> { + // Empty values are allowed in Kubernetes + if value.is_empty() { + return Ok(()); + } + + if value.len() > 63 { + return Err(Status::invalid_argument(format!( + "label value exceeds 63 characters: '{value}'" + ))); + } + + // Must contain only alphanumeric, hyphens, underscores, and dots + if !value + .chars() + .all(|c| c.is_alphanumeric() || c == '-' || c == '_' || c == '.') + { + return Err(Status::invalid_argument(format!( + "label value contains invalid characters (must be alphanumeric, '-', '_', or '.'): '{value}'" + ))); + } + + // Must start and end with alphanumeric + let first = value.chars().next().unwrap(); // safe: we checked !is_empty() + let last = value.chars().last().unwrap(); + if !first.is_alphanumeric() { + return Err(Status::invalid_argument(format!( + "label value must start with alphanumeric character: '{value}'" + ))); + } + if !last.is_alphanumeric() { + return Err(Status::invalid_argument(format!( + "label value must end with alphanumeric character: '{value}'" + ))); + } + + Ok(()) +} + +/// Validate a label selector string format. +/// +/// Format: "key1=value1,key2=value2" +/// Each key and value is validated using `validate_label_key` and `validate_label_value`. +/// Empty selectors are allowed. Trailing commas are ignored. +pub(super) fn validate_label_selector(selector: &str) -> Result<(), Status> { + if selector.trim().is_empty() { + return Ok(()); + } + + for pair in selector.split(',') { + let pair = pair.trim(); + if pair.is_empty() { + continue; + } + + let parts: Vec<&str> = pair.splitn(2, '=').collect(); + if parts.len() != 2 { + return Err(Status::invalid_argument(format!( + "invalid label selector: expected 'key=value', got '{pair}'" + ))); + } + + let key = parts[0].trim(); + let value = parts[1].trim(); + + if key.is_empty() { + return Err(Status::invalid_argument(format!( + "invalid label selector: key cannot be empty in '{pair}'" + ))); + } + + // Validate key and value using the Kubernetes-compliant validators + validate_label_key(key)?; + validate_label_value(value)?; + } + + Ok(()) +} + +// --------------------------------------------------------------------------- +// Object metadata validation +// --------------------------------------------------------------------------- + +/// Validate that object metadata is present and contains required fields. +/// +/// This ensures that all resources have valid metadata with non-empty ID and name, +/// preventing issues where missing metadata could lead to security vulnerabilities +/// (e.g., empty string IDs/names matching unintended resources). +/// +/// Returns `INVALID_ARGUMENT` if metadata is missing or invalid. +pub(super) fn validate_object_metadata( + metadata: Option<&openshell_core::proto::datamodel::v1::ObjectMeta>, + resource_type: &str, +) -> Result<(), Status> { + let metadata = metadata + .ok_or_else(|| Status::invalid_argument(format!("{resource_type} metadata is required")))?; + + if metadata.id.is_empty() { + return Err(Status::invalid_argument(format!( + "{resource_type} metadata.id cannot be empty" + ))); + } + + if metadata.name.is_empty() { + return Err(Status::invalid_argument(format!( + "{resource_type} metadata.name cannot be empty" + ))); + } + + // Validate all labels in metadata + for (key, value) in &metadata.labels { + validate_label_key(key)?; + validate_label_value(value)?; + } + + Ok(()) +} + // --------------------------------------------------------------------------- // Policy validation // --------------------------------------------------------------------------- @@ -381,7 +627,7 @@ pub(super) fn level_matches(log_level: &str, min_level: &str) -> bool { let to_num = |s: &str| match s.to_uppercase().as_str() { "ERROR" => 0, "WARN" => 1, - "INFO" => 2, + "INFO" | "OCSF" => 2, "DEBUG" => 3, "TRACE" => 4, _ => 5, // unknown levels always pass @@ -413,6 +659,12 @@ mod tests { SandboxSpec::default() } + #[test] + fn level_matches_treats_ocsf_as_info() { + assert!(level_matches("OCSF", "INFO")); + assert!(!level_matches("OCSF", "WARN")); + } + #[test] fn validate_sandbox_spec_accepts_gpu_flag() { let spec = SandboxSpec { @@ -610,28 +862,44 @@ mod tests { std::iter::once(("KEY".to_string(), "val".to_string())).collect() } + fn make_test_provider( + name: &str, + provider_type: &str, + credentials: HashMap, + config: HashMap, + ) -> Provider { + Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), + r#type: provider_type.to_string(), + credentials, + config, + } + } + #[test] fn validate_provider_fields_accepts_valid() { - let provider = Provider { - id: String::new(), - name: "my-provider".to_string(), - r#type: "claude".to_string(), - credentials: one_credential(), - config: std::iter::once(("endpoint".to_string(), "https://example.com".to_string())) - .collect(), - }; + let provider = make_test_provider( + "my-provider", + "claude", + one_credential(), + std::iter::once(("endpoint".to_string(), "https://example.com".to_string())).collect(), + ); assert!(validate_provider_fields(&provider).is_ok()); } #[test] fn validate_provider_fields_rejects_over_limit_name() { - let provider = Provider { - id: String::new(), - name: "a".repeat(MAX_NAME_LEN + 1), - r#type: "claude".to_string(), - credentials: one_credential(), - config: HashMap::new(), - }; + let provider = make_test_provider( + &"a".repeat(MAX_NAME_LEN + 1), + "claude", + one_credential(), + HashMap::new(), + ); let err = validate_provider_fields(&provider).unwrap_err(); assert_eq!(err.code(), Code::InvalidArgument); assert!(err.message().contains("provider.name")); @@ -639,13 +907,12 @@ mod tests { #[test] fn validate_provider_fields_rejects_over_limit_type() { - let provider = Provider { - id: String::new(), - name: "ok".to_string(), - r#type: "x".repeat(MAX_PROVIDER_TYPE_LEN + 1), - credentials: one_credential(), - config: HashMap::new(), - }; + let provider = make_test_provider( + "ok", + &"x".repeat(MAX_PROVIDER_TYPE_LEN + 1), + one_credential(), + HashMap::new(), + ); let err = validate_provider_fields(&provider).unwrap_err(); assert_eq!(err.code(), Code::InvalidArgument); assert!(err.message().contains("provider.type")); @@ -656,13 +923,7 @@ mod tests { let creds: HashMap = (0..=MAX_PROVIDER_CREDENTIALS_ENTRIES) .map(|i| (format!("K{i}"), "v".to_string())) .collect(); - let provider = Provider { - id: String::new(), - name: "ok".to_string(), - r#type: "claude".to_string(), - credentials: creds, - config: HashMap::new(), - }; + let provider = make_test_provider("ok", "claude", creds, HashMap::new()); let err = validate_provider_fields(&provider).unwrap_err(); assert_eq!(err.code(), Code::InvalidArgument); assert!(err.message().contains("provider.credentials")); @@ -673,13 +934,7 @@ mod tests { let config: HashMap = (0..=MAX_PROVIDER_CONFIG_ENTRIES) .map(|i| (format!("K{i}"), "v".to_string())) .collect(); - let provider = Provider { - id: String::new(), - name: "ok".to_string(), - r#type: "claude".to_string(), - credentials: one_credential(), - config, - }; + let provider = make_test_provider("ok", "claude", one_credential(), config); let err = validate_provider_fields(&provider).unwrap_err(); assert_eq!(err.code(), Code::InvalidArgument); assert!(err.message().contains("provider.config")); @@ -687,13 +942,12 @@ mod tests { #[test] fn validate_provider_fields_at_limit_name_accepted() { - let provider = Provider { - id: String::new(), - name: "a".repeat(MAX_NAME_LEN), - r#type: "claude".to_string(), - credentials: one_credential(), - config: HashMap::new(), - }; + let provider = make_test_provider( + &"a".repeat(MAX_NAME_LEN), + "claude", + one_credential(), + HashMap::new(), + ); assert!(validate_provider_fields(&provider).is_ok()); } @@ -701,13 +955,7 @@ mod tests { fn validate_provider_fields_rejects_oversized_credential_key() { let mut creds = HashMap::new(); creds.insert("k".repeat(MAX_MAP_KEY_LEN + 1), "v".to_string()); - let provider = Provider { - id: String::new(), - name: "ok".to_string(), - r#type: "claude".to_string(), - credentials: creds, - config: HashMap::new(), - }; + let provider = make_test_provider("ok", "claude", creds, HashMap::new()); let err = validate_provider_fields(&provider).unwrap_err(); assert_eq!(err.code(), Code::InvalidArgument); assert!(err.message().contains("key")); @@ -717,18 +965,300 @@ mod tests { fn validate_provider_fields_rejects_oversized_config_value() { let mut config = HashMap::new(); config.insert("k".to_string(), "v".repeat(MAX_MAP_VALUE_LEN + 1)); - let provider = Provider { - id: String::new(), - name: "ok".to_string(), - r#type: "claude".to_string(), - credentials: one_credential(), - config, - }; + let provider = make_test_provider("ok", "claude", one_credential(), config); let err = validate_provider_fields(&provider).unwrap_err(); assert_eq!(err.code(), Code::InvalidArgument); assert!(err.message().contains("value")); } + // ---- Label selector validation ---- + + // ---- Label key validation ---- + + #[test] + fn validate_label_key_accepts_simple_names() { + assert!(validate_label_key("app").is_ok()); + assert!(validate_label_key("my-app").is_ok()); + assert!(validate_label_key("my_app").is_ok()); + assert!(validate_label_key("my.app").is_ok()); + assert!(validate_label_key("app123").is_ok()); + assert!(validate_label_key("a1-b2_c3.d4").is_ok()); + } + + #[test] + fn validate_label_key_accepts_prefixed_names() { + assert!(validate_label_key("kubernetes.io/app").is_ok()); + assert!(validate_label_key("example.com/my-label").is_ok()); + assert!(validate_label_key("sub.domain.example.com/name").is_ok()); + assert!(validate_label_key("a.b/c").is_ok()); + } + + #[test] + fn validate_label_key_rejects_empty() { + let err = validate_label_key("").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("cannot be empty")); + } + + #[test] + fn validate_label_key_rejects_name_starting_with_hyphen() { + let err = validate_label_key("-app").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must start with alphanumeric")); + } + + #[test] + fn validate_label_key_rejects_name_ending_with_hyphen() { + let err = validate_label_key("app-").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must end with alphanumeric")); + } + + #[test] + fn validate_label_key_rejects_name_starting_with_underscore() { + let err = validate_label_key("_app").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must start with alphanumeric")); + } + + #[test] + fn validate_label_key_rejects_name_starting_with_dot() { + let err = validate_label_key(".app").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must start with alphanumeric")); + } + + #[test] + fn validate_label_key_rejects_name_too_long() { + let long_name = "a".repeat(64); + let err = validate_label_key(&long_name).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("exceeds 63 characters")); + } + + #[test] + fn validate_label_key_accepts_name_at_max_length() { + let max_name = format!("a{}z", "b".repeat(61)); + assert!(validate_label_key(&max_name).is_ok()); + } + + #[test] + fn validate_label_key_rejects_total_length_too_long() { + let long_key = format!("{}/app", "a".repeat(250)); + let err = validate_label_key(&long_key).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("exceeds 253 characters")); + } + + #[test] + fn validate_label_key_rejects_empty_prefix() { + let err = validate_label_key("/app").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("prefix cannot be empty")); + } + + #[test] + fn validate_label_key_rejects_empty_name_after_prefix() { + let err = validate_label_key("kubernetes.io/").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("name segment cannot be empty")); + } + + #[test] + fn validate_label_key_rejects_prefix_with_uppercase() { + let err = validate_label_key("Example.com/app").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must be a DNS subdomain")); + } + + #[test] + fn validate_label_key_rejects_prefix_starting_with_hyphen() { + let err = validate_label_key("-example.com/app").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!( + err.message() + .contains("cannot start or end with '-' or '.'") + ); + } + + #[test] + fn validate_label_key_rejects_prefix_ending_with_dot() { + let err = validate_label_key("example.com./app").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!( + err.message() + .contains("cannot start or end with '-' or '.'") + ); + } + + #[test] + fn validate_label_key_rejects_prefix_with_consecutive_dots() { + let err = validate_label_key("example..com/app").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("cannot contain consecutive dots")); + } + + #[test] + fn validate_label_key_rejects_invalid_characters() { + let err = validate_label_key("app@name").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("invalid characters")); + } + + // ---- Label value validation ---- + + #[test] + fn validate_label_value_accepts_empty() { + // Kubernetes allows empty label values + assert!(validate_label_value("").is_ok()); + } + + #[test] + fn validate_label_value_accepts_valid_values() { + assert!(validate_label_value("prod").is_ok()); + assert!(validate_label_value("my-value").is_ok()); + assert!(validate_label_value("my_value").is_ok()); + assert!(validate_label_value("my.value").is_ok()); + assert!(validate_label_value("value123").is_ok()); + assert!(validate_label_value("v1-2_3.4").is_ok()); + } + + #[test] + fn validate_label_value_accepts_max_length() { + let max_value = format!("a{}z", "b".repeat(61)); + assert!(validate_label_value(&max_value).is_ok()); + } + + #[test] + fn validate_label_value_rejects_too_long() { + let long_value = "a".repeat(64); + let err = validate_label_value(&long_value).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("exceeds 63 characters")); + } + + #[test] + fn validate_label_value_rejects_starting_with_hyphen() { + let err = validate_label_value("-value").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must start with alphanumeric")); + } + + #[test] + fn validate_label_value_rejects_ending_with_hyphen() { + let err = validate_label_value("value-").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must end with alphanumeric")); + } + + #[test] + fn validate_label_value_rejects_starting_with_underscore() { + let err = validate_label_value("_value").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must start with alphanumeric")); + } + + #[test] + fn validate_label_value_rejects_starting_with_dot() { + let err = validate_label_value(".value").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must start with alphanumeric")); + } + + #[test] + fn validate_label_value_rejects_invalid_characters() { + let err = validate_label_value("value@123").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("invalid characters")); + } + + // ---- Label selector validation ---- + + #[test] + fn validate_label_selector_accepts_empty() { + assert!(validate_label_selector("").is_ok()); + assert!(validate_label_selector(" ").is_ok()); + } + + #[test] + fn validate_label_selector_accepts_single_pair() { + assert!(validate_label_selector("env=prod").is_ok()); + assert!(validate_label_selector(" env=prod ").is_ok()); + } + + #[test] + fn validate_label_selector_accepts_multiple_pairs() { + assert!(validate_label_selector("env=prod,team=platform").is_ok()); + assert!(validate_label_selector("env=prod, team=platform").is_ok()); + } + + #[test] + fn validate_label_selector_rejects_missing_equals() { + let err = validate_label_selector("env:prod").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("expected 'key=value'")); + } + + #[test] + fn validate_label_selector_rejects_empty_key() { + let err = validate_label_selector("=prod").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("key cannot be empty")); + } + + #[test] + fn validate_label_selector_accepts_empty_value() { + // Kubernetes allows empty label values + assert!(validate_label_selector("env=").is_ok()); + assert!(validate_label_selector("app=,env=prod").is_ok()); + } + + #[test] + fn validate_label_selector_allows_trailing_comma() { + // Trailing commas are treated as empty pairs and ignored + assert!(validate_label_selector("env=prod,").is_ok()); + } + + #[test] + fn validate_label_selector_accepts_prefixed_keys() { + assert!(validate_label_selector("kubernetes.io/app=web").is_ok()); + assert!(validate_label_selector("example.com/env=prod,team=platform").is_ok()); + } + + #[test] + fn validate_label_selector_rejects_invalid_key_format() { + // Key starting with hyphen + let err = validate_label_selector("-app=prod").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must start with alphanumeric")); + } + + #[test] + fn validate_label_selector_rejects_invalid_value_format() { + // Value starting with hyphen + let err = validate_label_selector("env=-prod").unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must start with alphanumeric")); + } + + #[test] + fn validate_label_selector_rejects_oversized_key() { + let long_key = "a".repeat(64); + let selector = format!("{long_key}=value"); + let err = validate_label_selector(&selector).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("exceeds 63 characters")); + } + + #[test] + fn validate_label_selector_rejects_oversized_value() { + let long_value = "a".repeat(64); + let selector = format!("key={long_value}"); + let err = validate_label_selector(&selector).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("exceeds 63 characters")); + } + // ---- Policy safety ---- #[test] diff --git a/crates/openshell-server/src/http.rs b/crates/openshell-server/src/http.rs index afe7edc1b6..7650c2339a 100644 --- a/crates/openshell-server/src/http.rs +++ b/crates/openshell-server/src/http.rs @@ -3,7 +3,8 @@ //! HTTP health endpoints using Axum. -use axum::{Json, Router, http::StatusCode, response::IntoResponse, routing::get}; +use axum::{Json, Router, extract::State, http::StatusCode, response::IntoResponse, routing::get}; +use metrics_exporter_prometheus::PrometheusHandle; use serde::Serialize; use std::sync::Arc; @@ -45,10 +46,20 @@ pub fn health_router() -> Router { .route("/readyz", get(readyz)) } +/// Create the metrics router for the dedicated metrics port. +pub fn metrics_router(handle: PrometheusHandle) -> Router { + Router::new() + .route("/metrics", get(render_metrics)) + .with_state(handle) +} + +async fn render_metrics(State(handle): State) -> impl IntoResponse { + handle.render() +} + /// Create the HTTP router. pub fn http_router(state: Arc) -> Router { - health_router() - .merge(crate::ssh_tunnel::router(state.clone())) + crate::ssh_tunnel::router(state.clone()) .merge(crate::ws_tunnel::router(state.clone())) .merge(crate::auth::router(state)) } diff --git a/crates/openshell-server/src/inference.rs b/crates/openshell-server/src/inference.rs index 0fb29bde55..b52700f0d5 100644 --- a/crates/openshell-server/src/inference.rs +++ b/crates/openshell-server/src/inference.rs @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +#![allow(clippy::result_large_err)] // gRPC handlers return Result, Status> + use openshell_core::proto::{ ClusterInferenceConfig, GetClusterInferenceRequest, GetClusterInferenceResponse, GetInferenceBundleRequest, GetInferenceBundleResponse, InferenceRoute, Provider, ResolvedRoute, @@ -15,7 +17,7 @@ use tonic::{Request, Response, Status}; use crate::{ ServerState, - persistence::{ObjectId, ObjectName, ObjectType, Store}, + persistence::{ObjectName, ObjectType, Store, current_time_ms}, }; #[derive(Debug)] @@ -51,18 +53,6 @@ impl ObjectType for InferenceRoute { } } -impl ObjectId for InferenceRoute { - fn object_id(&self) -> &str { - &self.id - } -} - -impl ObjectName for InferenceRoute { - fn object_name(&self) -> &str { - &self.name - } -} - #[tonic::async_trait] impl Inference for InferenceService { async fn get_inference_bundle( @@ -172,7 +162,7 @@ async fn upsert_cluster_inference_route( let resolved = resolve_provider_route(&provider)?; let validation = if verify { - vec![verify_provider_endpoint(&provider.name, model_id, &resolved).await?] + vec![verify_provider_endpoint(provider.object_name(), model_id, &resolved).await?] } else { Vec::new() }; @@ -184,22 +174,31 @@ async fn upsert_cluster_inference_route( .await .map_err(|e| Status::internal(format!("fetch route failed: {e}")))?; + let now_ms = + current_time_ms().map_err(|e| Status::internal(format!("get current time: {e}")))?; + let route = if let Some(existing) = existing { InferenceRoute { - id: existing.id, - name: existing.name, + metadata: existing.metadata.clone(), config: Some(config), version: existing.version.saturating_add(1), } } else { InferenceRoute { - id: uuid::Uuid::new_v4().to_string(), - name: route_name.to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: uuid::Uuid::new_v4().to_string(), + name: route_name.to_string(), + created_at_ms: now_ms, + labels: std::collections::HashMap::new(), + }), config: Some(config), version: 1, } }; + // Ensure metadata is valid (defense in depth - should always be true for server-constructed metadata) + crate::grpc::validate_object_metadata(route.metadata.as_ref(), "inference_route")?; + store .put_message(&route) .await @@ -214,7 +213,7 @@ fn build_cluster_inference_config( timeout_secs: u64, ) -> ClusterInferenceConfig { ClusterInferenceConfig { - provider_name: provider.name.clone(), + provider_name: provider.object_name().to_string(), model_id: model_id.to_string(), timeout_secs, } @@ -238,7 +237,7 @@ fn resolve_provider_route(provider: &Provider) -> Result Result Result Result Result InferenceRoute { InferenceRoute { - id: format!("id-{name}"), - name: name.to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("id-{name}"), + name: name.to_string(), + created_at_ms: 1_000_000, + labels: std::collections::HashMap::new(), + }), config: Some(ClusterInferenceConfig { provider_name: provider_name.to_string(), model_id: model_id.to_string(), timeout_secs: 0, }), - version: 1, + version: 0, } } fn make_provider(name: &str, provider_type: &str, key_name: &str, key_value: &str) -> Provider { Provider { - id: format!("provider-{name}"), - name: name.to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("provider-{name}"), + name: name.to_string(), + created_at_ms: 1_000_000, + labels: std::collections::HashMap::new(), + }), r#type: provider_type.to_string(), credentials: std::iter::once((key_name.to_string(), key_value.to_string())).collect(), config: std::collections::HashMap::new(), @@ -534,8 +550,7 @@ mod tests { ) .await .expect("first set should succeed"); - assert_eq!(first.route.name, CLUSTER_INFERENCE_ROUTE_NAME); - assert_eq!(first.route.version, 1); + assert_eq!(first.route.object_name(), CLUSTER_INFERENCE_ROUTE_NAME); let second = upsert_cluster_inference_route( &store, @@ -547,8 +562,7 @@ mod tests { ) .await .expect("second set should succeed"); - assert_eq!(second.route.version, 2); - assert_eq!(second.route.id, first.route.id); + assert_eq!(second.route.object_id(), first.route.object_id()); let config = second.route.config.as_ref().expect("config"); assert_eq!(config.provider_name, "openai-dev"); @@ -647,8 +661,12 @@ mod tests { .expect("store should connect"); let provider = Provider { - id: "provider-1".to_string(), - name: "openai-dev".to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "provider-1".to_string(), + name: "openai-dev".to_string(), + created_at_ms: 1_000_000, + labels: std::collections::HashMap::new(), + }), r#type: "openai".to_string(), credentials: std::iter::once(("OPENAI_API_KEY".to_string(), "sk-test".to_string())) .collect(), @@ -664,14 +682,18 @@ mod tests { .expect("provider should persist"); let route = InferenceRoute { - id: "r-1".to_string(), - name: CLUSTER_INFERENCE_ROUTE_NAME.to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "r-1".to_string(), + name: CLUSTER_INFERENCE_ROUTE_NAME.to_string(), + created_at_ms: 1_000_000, + labels: std::collections::HashMap::new(), + }), config: Some(ClusterInferenceConfig { provider_name: "openai-dev".to_string(), model_id: "test/model".to_string(), timeout_secs: 0, }), - version: 7, + version: 1, }; store .put_message(&route) @@ -722,12 +744,11 @@ mod tests { assert_eq!(first.api_key, "sk-initial"); let rotated_provider = Provider { - id: provider.id, - name: provider.name, - r#type: provider.r#type, + metadata: provider.metadata.clone(), + r#type: provider.r#type.clone(), credentials: std::iter::once(("OPENAI_API_KEY".to_string(), "sk-rotated".to_string())) .collect(), - config: provider.config, + config: provider.config.clone(), }; store .put_message(&rotated_provider) @@ -761,8 +782,7 @@ mod tests { .await .expect("should succeed"); - assert_eq!(route.route.name, SANDBOX_SYSTEM_ROUTE_NAME); - assert_eq!(route.route.version, 1); + assert_eq!(route.route.object_name(), SANDBOX_SYSTEM_ROUTE_NAME); let config = route.route.config.as_ref().expect("config"); assert_eq!(config.provider_name, "anthropic-dev"); assert_eq!(config.model_id, "claude-sonnet-4-20250514"); @@ -854,7 +874,7 @@ mod tests { .expect("fetch should succeed") .expect("route should exist"); - assert_eq!(route.name, SANDBOX_SYSTEM_ROUTE_NAME); + assert_eq!(route.object_name(), SANDBOX_SYSTEM_ROUTE_NAME); let config = route.config.as_ref().expect("config"); assert_eq!(config.model_id, "gpt-4o-mini"); } diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index a8d820b4de..a80301c124 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -8,31 +8,48 @@ //! - HTTP health endpoints //! - Protocol multiplexing (gRPC + HTTP on same port) //! - mTLS support +//! +//! TODO(driver-abstraction): `build_compute_runtime` still switches on +//! [`ComputeDriverKind`] and calls driver-specific constructors +//! ([`ComputeRuntime::new_kubernetes`], [`compute::vm::spawn`] + +//! [`ComputeRuntime::new_remote_vm`]). Once we have a generalized compute +//! driver interface, the per-arm wiring here should collapse to a single +//! driver-agnostic path that asks each registered driver to produce a +//! [`Channel`](tonic::transport::Channel) and hands the rest of the gateway a +//! uniform [`ComputeRuntime`]. The remaining VM plumbing now lives in +//! [`compute::vm`]; keep this file driver-agnostic going forward. mod auth; +pub mod cli; mod compute; mod grpc; mod http; mod inference; mod multiplex; mod persistence; +pub(crate) mod policy_store; mod sandbox_index; mod sandbox_watch; mod ssh_tunnel; +pub mod supervisor_session; mod tls; pub mod tracing_bus; mod ws_tunnel; +use metrics_exporter_prometheus::PrometheusBuilder; use openshell_core::{ComputeDriverKind, Config, Error, Result}; use std::collections::HashMap; use std::io::ErrorKind; +use std::net::SocketAddr; use std::sync::{Arc, Mutex}; -use tokio::net::TcpListener; -use tracing::{debug, error, info}; +use std::time::Duration; +use tokio::net::{TcpListener, TcpStream}; +use tokio::sync::watch; +use tracing::{debug, error, info, warn}; -use compute::ComputeRuntime; +use compute::{ComputeRuntime, DockerComputeConfig, VmComputeConfig}; pub use grpc::OpenShellService; -pub use http::{health_router, http_router}; +pub use http::{health_router, http_router, metrics_router}; pub use multiplex::{MultiplexService, MultiplexedService}; use openshell_driver_kubernetes::KubernetesComputeConfig; use persistence::Store; @@ -73,6 +90,16 @@ pub struct ServerState { /// set/delete operation, including the precedence check on sandbox /// mutations that reads global state. pub settings_mutex: tokio::sync::Mutex<()>, + + /// Registry of active supervisor sessions and pending relay channels. + /// + /// Stored as `Arc` so compute drivers (e.g. the Docker driver) + /// can be constructed before `ServerState` and still + /// query session state to surface supervisor readiness. + pub supervisor_sessions: Arc, + + /// OIDC JWKS cache for JWT validation. `None` when OIDC is not configured. + pub oidc_cache: Option>, } fn is_benign_tls_handshake_failure(error: &std::io::Error) -> bool { @@ -85,6 +112,7 @@ fn is_benign_tls_handshake_failure(error: &std::io::Error) -> bool { impl ServerState { /// Create new server state. #[must_use] + #[allow(clippy::too_many_arguments)] pub fn new( config: Config, store: Arc, @@ -92,6 +120,8 @@ impl ServerState { sandbox_index: SandboxIndex, sandbox_watch_bus: SandboxWatchBus, tracing_log_bus: TracingLogBus, + supervisor_sessions: Arc, + oidc_cache: Option>, ) -> Self { Self { config, @@ -103,6 +133,8 @@ impl ServerState { ssh_connections_by_token: Mutex::new(HashMap::new()), ssh_connections_by_sandbox: Mutex::new(HashMap::new()), settings_mutex: tokio::sync::Mutex::new(()), + supervisor_sessions, + oidc_cache, } } } @@ -114,12 +146,20 @@ impl ServerState { /// # Errors /// /// Returns an error if the server fails to start or encounters a fatal error. -pub async fn run_server(config: Config, tracing_log_bus: TracingLogBus) -> Result<()> { +pub async fn run_server( + config: Config, + vm_config: VmComputeConfig, + docker_config: DockerComputeConfig, + tracing_log_bus: TracingLogBus, +) -> Result<()> { let database_url = config.database_url.trim(); if database_url.is_empty() { return Err(Error::config("database_url is required")); } - if config.ssh_handshake_secret.is_empty() { + let driver = configured_compute_driver(&config)?; + if config.ssh_handshake_secret.is_empty() + && !matches!(driver, ComputeDriverKind::Docker | ComputeDriverKind::Vm) + { return Err(Error::config( "ssh_handshake_secret is required. Set --ssh-handshake-secret or OPENSHELL_SSH_HANDSHAKE_SECRET", )); @@ -127,14 +167,36 @@ pub async fn run_server(config: Config, tracing_log_bus: TracingLogBus) -> Resul let store = Arc::new(Store::connect(database_url).await?); + let oidc_cache = if let Some(ref oidc) = config.oidc { + // Validate RBAC configuration before starting. + let policy = auth::authz::AuthzPolicy { + admin_role: oidc.admin_role.clone(), + user_role: oidc.user_role.clone(), + scopes_enabled: !oidc.scopes_claim.is_empty(), + }; + policy.validate().map_err(Error::config)?; + + let cache = auth::oidc::JwksCache::new(oidc) + .await + .map_err(|e| Error::config(format!("OIDC initialization failed: {e}")))?; + info!("OIDC JWT validation enabled (issuer: {})", oidc.issuer); + Some(Arc::new(cache)) + } else { + None + }; + let sandbox_index = SandboxIndex::new(); let sandbox_watch_bus = SandboxWatchBus::new(); + let supervisor_sessions = Arc::new(supervisor_session::SupervisorSessionRegistry::new()); let compute = build_compute_runtime( &config, + &vm_config, + &docker_config, store.clone(), sandbox_index.clone(), sandbox_watch_bus.clone(), tracing_log_bus.clone(), + supervisor_sessions.clone(), ) .await?; let state = Arc::new(ServerState::new( @@ -144,20 +206,81 @@ pub async fn run_server(config: Config, tracing_log_bus: TracingLogBus) -> Resul sandbox_index, sandbox_watch_bus, tracing_log_bus, + supervisor_sessions, + oidc_cache, )); + // Resume sandboxes that were stopped during the previous gateway + // shutdown so the running compute state matches the persisted store. + // Runs before watchers spawn so the watch loop sees the post-resume + // snapshot on its first poll. + if let Err(err) = state.compute.resume_persisted_sandboxes().await { + warn!(error = %err, "Failed to resume persisted sandboxes during startup"); + } + state.compute.spawn_watchers(); - ssh_tunnel::spawn_session_reaper(store.clone(), std::time::Duration::from_secs(3600)); + ssh_tunnel::spawn_session_reaper(store.clone(), Duration::from_secs(3600)); + supervisor_session::spawn_relay_reaper(state.clone(), Duration::from_secs(30)); // Create the multiplexed service let service = MultiplexService::new(state.clone()); - // Bind the TCP listener - let listener = TcpListener::bind(config.bind_address) - .await - .map_err(|e| Error::transport(format!("failed to bind to {}: {e}", config.bind_address)))?; + let mut extra_listener_addresses = config.extra_bind_addresses.clone(); + extra_listener_addresses.extend_from_slice(state.compute.gateway_bind_addresses()); + let gateway_listener_addresses = + gateway_listener_addresses(config.bind_address, &extra_listener_addresses); + let mut gateway_listeners = Vec::with_capacity(gateway_listener_addresses.len()); + for address in gateway_listener_addresses { + let listener = TcpListener::bind(address) + .await + .map_err(|e| Error::transport(format!("failed to bind to {address}: {e}")))?; + let local_addr = listener.local_addr().unwrap_or(address); + info!(address = %local_addr, "Server listening"); + gateway_listeners.push((listener, local_addr)); + } - info!(address = %config.bind_address, "Server listening"); + // Bind the unauthenticated health endpoint on a separate port when configured. + if let Some(health_bind_address) = config.health_bind_address { + let health_listener = TcpListener::bind(health_bind_address).await.map_err(|e| { + Error::transport(format!( + "failed to bind health port {health_bind_address}: {e}" + )) + })?; + info!(address = %health_bind_address, "Health server listening"); + tokio::spawn(async move { + if let Err(e) = axum::serve(health_listener, health_router().into_make_service()).await + { + error!("Health server error: {e}"); + } + }); + } else { + info!("Health server disabled"); + } + + // Bind the Prometheus metrics endpoint on a dedicated port when configured. + if let Some(metrics_bind_address) = config.metrics_bind_address { + let prometheus_handle = PrometheusBuilder::new() + .install_recorder() + .map_err(|e| Error::config(format!("failed to install metrics recorder: {e}")))?; + let metrics_listener = TcpListener::bind(metrics_bind_address).await.map_err(|e| { + Error::transport(format!( + "failed to bind metrics port {metrics_bind_address}: {e}", + )) + })?; + info!(address = %metrics_bind_address, "Metrics server listening"); + tokio::spawn(async move { + if let Err(e) = axum::serve( + metrics_listener, + metrics_router(prometheus_handle).into_make_service(), + ) + .await + { + error!("Metrics server error: {e}"); + } + }); + } else { + info!("Metrics server disabled"); + } // Build TLS acceptor when TLS is configured; otherwise serve plaintext. let tls_acceptor = if let Some(tls) = &config.tls { @@ -172,92 +295,328 @@ pub async fn run_server(config: Config, tracing_log_bus: TracingLogBus) -> Resul None }; - // Accept connections + let (shutdown_tx, shutdown_rx) = watch::channel(false); + let mut listener_tasks = Vec::with_capacity(gateway_listeners.len()); + for (listener, listen_addr) in gateway_listeners { + listener_tasks.push(tokio::spawn(serve_gateway_listener( + listener, + listen_addr, + service.clone(), + tls_acceptor.clone(), + shutdown_rx.clone(), + ))); + } + + shutdown_signal().await; + info!("Shutdown signal received; stopping gateway"); + let _ = shutdown_tx.send(true); + + for task in listener_tasks { + if let Err(err) = task.await { + warn!(error = %err, "Gateway listener task failed during shutdown"); + } + } + + state + .compute + .cleanup_on_shutdown() + .await + .map_err(|err| Error::execution(format!("gateway shutdown cleanup failed: {err}")))?; + + Ok(()) +} + +fn gateway_listener_addresses( + bind_address: SocketAddr, + extra_addresses: &[SocketAddr], +) -> Vec { + let mut addresses = vec![bind_address]; + for address in extra_addresses { + if !addresses + .iter() + .any(|existing| listener_covers(*existing, *address)) + { + addresses.push(*address); + } + } + addresses +} + +fn listener_covers(existing: SocketAddr, requested: SocketAddr) -> bool { + if existing == requested { + return true; + } + if existing.port() != requested.port() { + return false; + } + + match (existing.ip(), requested.ip()) { + (std::net::IpAddr::V4(existing), std::net::IpAddr::V4(_)) => existing.is_unspecified(), + (std::net::IpAddr::V6(existing), std::net::IpAddr::V6(_)) => existing.is_unspecified(), + _ => false, + } +} + +async fn serve_gateway_listener( + listener: TcpListener, + listen_addr: SocketAddr, + service: MultiplexService, + tls_acceptor: Option, + mut shutdown: watch::Receiver, +) { loop { - let (stream, addr) = match listener.accept().await { + let accepted = tokio::select! { + changed = shutdown.changed() => { + if changed.is_err() || *shutdown.borrow() { + break; + } + continue; + } + accepted = listener.accept() => accepted, + }; + + let (stream, addr) = match accepted { Ok(conn) => conn, Err(e) => { - error!(error = %e, "Failed to accept connection"); + error!(error = %e, listen = %listen_addr, "Failed to accept connection"); continue; } }; - let service = service.clone(); + spawn_gateway_connection(stream, addr, service.clone(), tls_acceptor.clone()); + } +} - if let Some(ref acceptor) = tls_acceptor { - let tls_acceptor = acceptor.clone(); - tokio::spawn(async move { - match tls_acceptor.inner().accept(stream).await { - Ok(tls_stream) => { - if let Err(e) = service.serve(tls_stream).await { - error!(error = %e, client = %addr, "Connection error"); - } - } - Err(e) => { - if is_benign_tls_handshake_failure(&e) { - debug!(error = %e, client = %addr, "TLS handshake closed early"); - } else { - error!(error = %e, client = %addr, "TLS handshake failed"); - } +fn spawn_gateway_connection( + stream: TcpStream, + addr: SocketAddr, + service: MultiplexService, + tls_acceptor: Option, +) { + if let Some(acceptor) = tls_acceptor { + tokio::spawn(async move { + match acceptor.inner().accept(stream).await { + Ok(tls_stream) => { + if let Err(e) = service.serve(tls_stream).await { + error!(error = %e, client = %addr, "Connection error"); } } - }); - } else { - tokio::spawn(async move { - if let Err(e) = service.serve(stream).await { - error!(error = %e, client = %addr, "Connection error"); + Err(e) => { + if is_benign_tls_handshake_failure(&e) { + debug!(error = %e, client = %addr, "TLS handshake closed early"); + } else { + error!(error = %e, client = %addr, "TLS handshake failed"); + } } - }); + } + }); + } else { + tokio::spawn(async move { + if let Err(e) = service.serve(stream).await { + error!(error = %e, client = %addr, "Connection error"); + } + }); + } +} + +async fn shutdown_signal() { + #[cfg(unix)] + { + tokio::select! { + () = ctrl_c_signal() => {} + () = terminate_signal() => {} } } + + #[cfg(not(unix))] + { + ctrl_c_signal().await; + } } +async fn ctrl_c_signal() { + if let Err(err) = tokio::signal::ctrl_c().await { + warn!(error = %err, "Failed to install Ctrl-C signal handler"); + std::future::pending::<()>().await; + } +} + +#[cfg(unix)] +async fn terminate_signal() { + let Ok(mut signal) = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) + else { + warn!("Failed to install SIGTERM signal handler"); + std::future::pending::<()>().await; + return; + }; + let _ = signal.recv().await; +} + +// Internal wiring helper: each argument is a distinct piece of runtime state +// that must be passed through, so the count is justified. +#[allow(clippy::too_many_arguments)] async fn build_compute_runtime( config: &Config, + vm_config: &VmComputeConfig, + docker_config: &DockerComputeConfig, store: Arc, sandbox_index: SandboxIndex, sandbox_watch_bus: SandboxWatchBus, tracing_log_bus: TracingLogBus, + supervisor_sessions: Arc, ) -> Result { let driver = configured_compute_driver(config)?; info!(driver = %driver, "Using compute driver"); match driver { - ComputeDriverKind::Kubernetes => ComputeRuntime::new_kubernetes( - KubernetesComputeConfig { - namespace: config.sandbox_namespace.clone(), - default_image: config.sandbox_image.clone(), - image_pull_policy: config.sandbox_image_pull_policy.clone(), - grpc_endpoint: config.grpc_endpoint.clone(), - ssh_listen_addr: format!("0.0.0.0:{}", config.sandbox_ssh_port), - ssh_port: config.sandbox_ssh_port, - ssh_handshake_secret: config.ssh_handshake_secret.clone(), - ssh_handshake_skew_secs: config.ssh_handshake_skew_secs, - client_tls_secret_name: config.client_tls_secret_name.clone(), - host_gateway_ip: config.host_gateway_ip.clone(), - }, + ComputeDriverKind::Kubernetes => { + let supervisor_image = std::env::var("OPENSHELL_SUPERVISOR_IMAGE") + .ok() + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| openshell_core::config::DEFAULT_SUPERVISOR_IMAGE.to_string()); + let supervisor_image_pull_policy = + std::env::var("OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY") + .ok() + .filter(|s| !s.is_empty()) + .unwrap_or_default(); + ComputeRuntime::new_kubernetes( + KubernetesComputeConfig { + namespace: config.sandbox_namespace.clone(), + default_image: config.sandbox_image.clone(), + image_pull_policy: config.sandbox_image_pull_policy.clone(), + supervisor_image, + supervisor_image_pull_policy, + grpc_endpoint: config.grpc_endpoint.clone(), + // Filesystem path to the supervisor's Unix-socket SSH daemon. + // The path lives in a root-only directory so only the + // supervisor can connect; the gateway reaches it through the + // RelayStream bridge, not directly. Override via + // `sandbox_ssh_socket_path` in the config for deployments + // where multiple supervisors share a filesystem. + ssh_socket_path: config.sandbox_ssh_socket_path.clone(), + ssh_handshake_secret: config.ssh_handshake_secret.clone(), + ssh_handshake_skew_secs: config.ssh_handshake_skew_secs, + client_tls_secret_name: config.client_tls_secret_name.clone(), + host_gateway_ip: config.host_gateway_ip.clone(), + }, + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions.clone(), + ) + .await + .map_err(|e| Error::execution(format!("failed to create compute runtime: {e}"))) + } + ComputeDriverKind::Docker => ComputeRuntime::new_docker( + config.clone(), + docker_config.clone(), store, sandbox_index, sandbox_watch_bus, tracing_log_bus, + supervisor_sessions, ) .await .map_err(|e| Error::execution(format!("failed to create compute runtime: {e}"))), - ComputeDriverKind::Podman => Err(Error::config( - "compute driver 'podman' is not implemented yet", - )), + ComputeDriverKind::Vm => { + let (channel, driver_process) = compute::vm::spawn(config, vm_config).await?; + ComputeRuntime::new_remote_vm( + channel, + Some(driver_process), + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions, + ) + .await + .map_err(|e| Error::execution(format!("failed to create compute runtime: {e}"))) + } + ComputeDriverKind::Podman => { + let socket_path = std::env::var("OPENSHELL_PODMAN_SOCKET") + .ok() + .filter(|s| !s.is_empty()) + .map_or_else( + openshell_driver_podman::PodmanComputeConfig::default_socket_path, + std::path::PathBuf::from, + ); + + let network_name = std::env::var("OPENSHELL_NETWORK_NAME") + .ok() + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| openshell_core::config::DEFAULT_NETWORK_NAME.to_string()); + + let stop_timeout_secs: u32 = std::env::var("OPENSHELL_STOP_TIMEOUT") + .ok() + .and_then(|s| s.parse().ok()) + .unwrap_or(openshell_core::config::DEFAULT_STOP_TIMEOUT_SECS); + + let supervisor_image = std::env::var("OPENSHELL_SUPERVISOR_IMAGE") + .ok() + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| openshell_core::config::DEFAULT_SUPERVISOR_IMAGE.to_string()); + + // TLS client cert paths for sandbox mTLS. When all three are + // set, the Podman driver bind-mounts them into sandbox + // containers and switches the endpoint to https://. + let podman_tls_ca = std::env::var("OPENSHELL_PODMAN_TLS_CA") + .ok() + .filter(|s| !s.is_empty()) + .map(std::path::PathBuf::from); + let podman_tls_cert = std::env::var("OPENSHELL_PODMAN_TLS_CERT") + .ok() + .filter(|s| !s.is_empty()) + .map(std::path::PathBuf::from); + let podman_tls_key = std::env::var("OPENSHELL_PODMAN_TLS_KEY") + .ok() + .filter(|s| !s.is_empty()) + .map(std::path::PathBuf::from); + + ComputeRuntime::new_podman( + openshell_driver_podman::PodmanComputeConfig { + socket_path, + default_image: config.sandbox_image.clone(), + image_pull_policy: config.sandbox_image_pull_policy.parse().unwrap_or_default(), + grpc_endpoint: config.grpc_endpoint.clone(), + gateway_port: config.bind_address.port(), + sandbox_ssh_socket_path: config.sandbox_ssh_socket_path.clone(), + network_name, + ssh_port: config.sandbox_ssh_port, + ssh_handshake_secret: config.ssh_handshake_secret.clone(), + ssh_handshake_skew_secs: config.ssh_handshake_skew_secs, + stop_timeout_secs, + supervisor_image, + guest_tls_ca: podman_tls_ca, + guest_tls_cert: podman_tls_cert, + guest_tls_key: podman_tls_key, + }, + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions, + ) + .await + .map_err(|e| Error::execution(format!("failed to create compute runtime: {e}"))) + } } } fn configured_compute_driver(config: &Config) -> Result { match config.compute_drivers.as_slice() { - [] => Err(Error::config( - "at least one compute driver must be configured", - )), - [driver @ ComputeDriverKind::Kubernetes] => Ok(*driver), - [ComputeDriverKind::Podman] => Err(Error::config( - "compute driver 'podman' is not implemented yet", - )), + [] => openshell_core::config::detect_driver().ok_or_else(|| { + Error::config( + "no compute driver configured and auto-detection found no suitable driver; \ + set --drivers or OPENSHELL_DRIVERS to kubernetes, podman, docker, or vm", + ) + }), + [ + driver @ (ComputeDriverKind::Kubernetes + | ComputeDriverKind::Vm + | ComputeDriverKind::Docker + | ComputeDriverKind::Podman), + ] => Ok(*driver), drivers => Err(Error::config(format!( "multiple compute drivers are not supported yet; configured drivers: {}", drivers @@ -271,9 +630,12 @@ fn configured_compute_driver(config: &Config) -> Result { #[cfg(test)] mod tests { - use super::{configured_compute_driver, is_benign_tls_handshake_failure}; + use super::{ + configured_compute_driver, gateway_listener_addresses, is_benign_tls_handshake_failure, + }; use openshell_core::{ComputeDriverKind, Config}; use std::io::{Error, ErrorKind}; + use std::net::SocketAddr; #[test] fn classifies_probe_style_tls_disconnects_as_benign() { @@ -296,18 +658,33 @@ mod tests { } #[test] - fn configured_compute_driver_defaults_to_kubernetes() { - assert_eq!( - configured_compute_driver(&Config::new(None)).unwrap(), - ComputeDriverKind::Kubernetes - ); - } - - #[test] - fn configured_compute_driver_requires_at_least_one_entry() { + fn configured_compute_driver_triggers_auto_detection_when_empty() { let config = Config::new(None).with_compute_drivers([]); - let err = configured_compute_driver(&config).unwrap_err(); - assert!(err.to_string().contains("at least one compute driver")); + // Empty drivers triggers auto-detection, which may return Some or None + // depending on the environment. This test verifies the auto-detection path + // is taken rather than immediately returning an error. + let result = configured_compute_driver(&config); + // Either we get a detected driver or an error about none being detected + match result { + Ok(driver) => { + assert!( + matches!( + driver, + ComputeDriverKind::Kubernetes + | ComputeDriverKind::Docker + | ComputeDriverKind::Podman + ), + "auto-detected unexpected driver: {driver:?}" + ); + } + Err(e) => { + assert!( + e.to_string() + .contains("no compute driver configured and none detected"), + "unexpected error: {e}" + ); + } + } } #[test] @@ -323,12 +700,51 @@ mod tests { } #[test] - fn configured_compute_driver_rejects_unimplemented_driver() { + fn configured_compute_driver_accepts_podman() { let config = Config::new(None).with_compute_drivers([ComputeDriverKind::Podman]); - let err = configured_compute_driver(&config).unwrap_err(); - assert!( - err.to_string() - .contains("compute driver 'podman' is not implemented yet") + assert_eq!( + configured_compute_driver(&config).unwrap(), + ComputeDriverKind::Podman + ); + } + + #[test] + fn configured_compute_driver_accepts_vm() { + let config = Config::new(None).with_compute_drivers([ComputeDriverKind::Vm]); + assert_eq!( + configured_compute_driver(&config).unwrap(), + ComputeDriverKind::Vm + ); + } + + #[test] + fn configured_compute_driver_accepts_docker() { + let config = Config::new(None).with_compute_drivers([ComputeDriverKind::Docker]); + assert_eq!( + configured_compute_driver(&config).unwrap(), + ComputeDriverKind::Docker + ); + } + + #[test] + fn gateway_listener_addresses_skip_driver_address_covered_by_wildcard() { + let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap(); + let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap(); + + assert_eq!( + gateway_listener_addresses(primary, &[docker, docker]), + vec![primary] + ); + } + + #[test] + fn gateway_listener_addresses_include_driver_address_on_distinct_ip() { + let primary: SocketAddr = "127.0.0.1:8080".parse().unwrap(); + let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap(); + + assert_eq!( + gateway_listener_addresses(primary, &[docker, docker]), + vec![primary, docker] ); } } diff --git a/crates/openshell-server/src/main.rs b/crates/openshell-server/src/main.rs index ed6c73825a..0f33c685f4 100644 --- a/crates/openshell-server/src/main.rs +++ b/crates/openshell-server/src/main.rs @@ -1,221 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! `OpenShell` Server - gRPC/HTTP server with protocol multiplexing. +//! `OpenShell` Gateway binary entrypoint. -use clap::Parser; -use miette::{IntoDiagnostic, Result}; -use openshell_core::ComputeDriverKind; -use std::net::SocketAddr; -use std::path::PathBuf; -use tracing::info; -use tracing_subscriber::EnvFilter; - -use openshell_server::{run_server, tracing_bus::TracingLogBus}; - -/// `OpenShell` Server - gRPC and HTTP server with protocol multiplexing. -#[derive(Parser, Debug)] -#[command(name = "openshell-server")] -#[command(version = openshell_core::VERSION)] -#[command(about = "OpenShell gRPC/HTTP server", long_about = None)] -struct Args { - /// Port to bind the server to (all interfaces). - #[arg(long, default_value_t = 8080, env = "OPENSHELL_SERVER_PORT")] - port: u16, - - /// Log level (trace, debug, info, warn, error). - #[arg(long, default_value = "info", env = "OPENSHELL_LOG_LEVEL")] - log_level: String, - - /// Path to TLS certificate file (required unless --disable-tls). - #[arg(long, env = "OPENSHELL_TLS_CERT")] - tls_cert: Option, - - /// Path to TLS private key file (required unless --disable-tls). - #[arg(long, env = "OPENSHELL_TLS_KEY")] - tls_key: Option, - - /// Path to CA certificate for client certificate verification (mTLS). - #[arg(long, env = "OPENSHELL_TLS_CLIENT_CA")] - tls_client_ca: Option, - - /// Database URL for persistence. - #[arg(long, env = "OPENSHELL_DB_URL", required = true)] - db_url: String, - - /// Compute drivers configured for this gateway. - /// - /// Accepts a comma-delimited list such as `kubernetes` or - /// `kubernetes,podman`. The configuration format is future-proofed for - /// multiple drivers, but the gateway currently requires exactly one. - #[arg( - long, - alias = "driver", - env = "OPENSHELL_DRIVERS", - value_delimiter = ',', - default_value = "kubernetes", - value_parser = parse_compute_driver - )] - drivers: Vec, - - /// Kubernetes namespace for sandboxes. - #[arg(long, env = "OPENSHELL_SANDBOX_NAMESPACE", default_value = "default")] - sandbox_namespace: String, - - /// Default container image for sandboxes. - #[arg(long, env = "OPENSHELL_SANDBOX_IMAGE")] - sandbox_image: Option, - - /// Kubernetes imagePullPolicy for sandbox pods (Always, IfNotPresent, Never). - #[arg(long, env = "OPENSHELL_SANDBOX_IMAGE_PULL_POLICY")] - sandbox_image_pull_policy: Option, - - /// gRPC endpoint for sandboxes to callback to `OpenShell`. - /// This should be reachable from within the Kubernetes cluster. - #[arg(long, env = "OPENSHELL_GRPC_ENDPOINT")] - grpc_endpoint: Option, - - /// Public host for the SSH gateway. - #[arg(long, env = "OPENSHELL_SSH_GATEWAY_HOST", default_value = "127.0.0.1")] - ssh_gateway_host: String, - - /// Public port for the SSH gateway. - #[arg(long, env = "OPENSHELL_SSH_GATEWAY_PORT", default_value_t = 8080)] - ssh_gateway_port: u16, - - /// HTTP path for SSH CONNECT/upgrade. - #[arg( - long, - env = "OPENSHELL_SSH_CONNECT_PATH", - default_value = "/connect/ssh" - )] - ssh_connect_path: String, - - /// SSH port inside sandbox pods. - #[arg(long, env = "OPENSHELL_SANDBOX_SSH_PORT", default_value_t = 2222)] - sandbox_ssh_port: u16, - - /// Shared secret for gateway-to-sandbox SSH handshake. - #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SECRET")] - ssh_handshake_secret: Option, - - /// Allowed clock skew in seconds for SSH handshake. - #[arg(long, env = "OPENSHELL_SSH_HANDSHAKE_SKEW_SECS", default_value_t = 300)] - ssh_handshake_skew_secs: u64, - - /// Kubernetes secret name containing client TLS materials for sandbox pods. - #[arg(long, env = "OPENSHELL_CLIENT_TLS_SECRET_NAME")] - client_tls_secret_name: Option, - - /// Host gateway IP for sandbox pod hostAliases. - /// When set, sandbox pods get hostAliases entries mapping - /// host.docker.internal and host.openshell.internal to this IP. - #[arg(long, env = "OPENSHELL_HOST_GATEWAY_IP")] - host_gateway_ip: Option, - - /// Disable TLS entirely — listen on plaintext HTTP. - /// Use this when the gateway sits behind a reverse proxy or tunnel - /// (e.g. Cloudflare Tunnel) that terminates TLS at the edge. - #[arg(long, env = "OPENSHELL_DISABLE_TLS")] - disable_tls: bool, - - /// Disable gateway authentication (mTLS client certificate requirement). - /// When set, the TLS handshake accepts connections without a client - /// certificate. Ignored when --disable-tls is set. - #[arg(long, env = "OPENSHELL_DISABLE_GATEWAY_AUTH")] - disable_gateway_auth: bool, -} +use miette::Result; #[tokio::main] async fn main() -> Result<()> { - rustls::crypto::ring::default_provider() - .install_default() - .map_err(|e| miette::miette!("failed to install rustls crypto provider: {e:?}"))?; - - let args = Args::parse(); - - // Initialize tracing - let tracing_log_bus = TracingLogBus::new(); - tracing_log_bus.install_subscriber( - EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)), - ); - - // Build configuration - let bind = SocketAddr::from(([0, 0, 0, 0], args.port)); - - let tls = if args.disable_tls { - None - } else { - let cert_path = args.tls_cert.ok_or_else(|| { - miette::miette!( - "--tls-cert is required when TLS is enabled (use --disable-tls to skip)" - ) - })?; - let key_path = args.tls_key.ok_or_else(|| { - miette::miette!("--tls-key is required when TLS is enabled (use --disable-tls to skip)") - })?; - let client_ca_path = args.tls_client_ca.ok_or_else(|| { - miette::miette!( - "--tls-client-ca is required when TLS is enabled (use --disable-tls to skip)" - ) - })?; - Some(openshell_core::TlsConfig { - cert_path, - key_path, - client_ca_path, - allow_unauthenticated: args.disable_gateway_auth, - }) - }; - - let mut config = openshell_core::Config::new(tls) - .with_bind_address(bind) - .with_log_level(&args.log_level); - - config = config - .with_database_url(args.db_url) - .with_compute_drivers(args.drivers) - .with_sandbox_namespace(args.sandbox_namespace) - .with_ssh_gateway_host(args.ssh_gateway_host) - .with_ssh_gateway_port(args.ssh_gateway_port) - .with_ssh_connect_path(args.ssh_connect_path) - .with_sandbox_ssh_port(args.sandbox_ssh_port) - .with_ssh_handshake_skew_secs(args.ssh_handshake_skew_secs); - - if let Some(image) = args.sandbox_image { - config = config.with_sandbox_image(image); - } - - if let Some(policy) = args.sandbox_image_pull_policy { - config = config.with_sandbox_image_pull_policy(policy); - } - - if let Some(endpoint) = args.grpc_endpoint { - config = config.with_grpc_endpoint(endpoint); - } - - if let Some(secret) = args.ssh_handshake_secret { - config = config.with_ssh_handshake_secret(secret); - } - - if let Some(name) = args.client_tls_secret_name { - config = config.with_client_tls_secret_name(name); - } - - if let Some(ip) = args.host_gateway_ip { - config = config.with_host_gateway_ip(ip); - } - - if args.disable_tls { - info!("TLS disabled — listening on plaintext HTTP"); - } else if args.disable_gateway_auth { - info!("Gateway auth disabled — accepting connections without client certificates"); - } - - info!(bind = %config.bind_address, "Starting OpenShell server"); - - run_server(config, tracing_log_bus).await.into_diagnostic() -} - -fn parse_compute_driver(value: &str) -> std::result::Result { - value.parse() + openshell_server::cli::run_cli().await } diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index 5ba44b1ec0..93e58d2025 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -7,7 +7,7 @@ //! to either the gRPC service or HTTP endpoints based on the request headers. use bytes::Bytes; -use http::{Request, Response}; +use http::{HeaderValue, Request, Response}; use http_body::Body; use http_body_util::BodyExt; use hyper::body::Incoming; @@ -15,6 +15,7 @@ use hyper_util::{ rt::{TokioExecutor, TokioIo}, server::conn::auto::Builder, }; +use metrics::{counter, histogram}; use openshell_core::proto::{ inference_server::InferenceServer, open_shell_server::OpenShellServer, }; @@ -22,10 +23,87 @@ use std::future::Future; use std::pin::Pin; use std::sync::Arc; use std::task::{Context, Poll}; +use std::time::{Duration, Instant}; use tokio::io::{AsyncRead, AsyncWrite}; use tower::ServiceExt; +use tower_http::request_id::{MakeRequestId, RequestId}; +use tracing::Span; -use crate::{OpenShellService, ServerState, http_router, inference::InferenceService}; +use crate::{ + OpenShellService, ServerState, auth::authz::AuthzPolicy, auth::oidc, http_router, + inference::InferenceService, +}; + +/// Request-ID generator that produces a UUID v4 for each inbound request. +#[derive(Clone)] +struct UuidRequestId; + +impl MakeRequestId for UuidRequestId { + fn make_request_id(&mut self, _req: &Request) -> Option { + let id = uuid::Uuid::new_v4().to_string(); + Some(RequestId::new(HeaderValue::from_str(&id).unwrap())) + } +} + +/// Build a tracing span for an inbound request, recording the `request_id` +/// header (set by [`UuidRequestId`] or supplied by the client). +fn make_request_span(req: &Request) -> Span { + let path = req.uri().path(); + let request_id = req + .headers() + .get("x-request-id") + .and_then(|v| v.to_str().ok()) + .unwrap_or("-"); + + if matches!(path, "/health" | "/healthz" | "/readyz") { + tracing::debug_span!( + "request", + method = %req.method(), + path, + request_id, + ) + } else { + tracing::info_span!( + "request", + method = %req.method(), + path, + request_id, + ) + } +} + +/// Log response status and latency within the request span. +fn log_response(res: &Response, latency: Duration, _span: &Span) { + tracing::info!( + status = res.status().as_u16(), + latency_ms = latency.as_millis(), + "response" + ); +} + +/// Wrap a service with the standard request-ID middleware stack. +/// +/// Layer order: `SetRequestId` → `TraceLayer` → `PropagateRequestId`. +macro_rules! request_id_middleware { + ($service:expr) => {{ + let x_request_id = ::http::HeaderName::from_static("x-request-id"); + ::tower::ServiceBuilder::new() + .layer(::tower_http::request_id::SetRequestIdLayer::new( + x_request_id.clone(), + UuidRequestId, + )) + .layer( + ::tower_http::trace::TraceLayer::new_for_http() + .make_span_with(make_request_span) + .on_request(()) + .on_response(log_response), + ) + .layer(::tower_http::request_id::PropagateRequestIdLayer::new( + x_request_id, + )) + .service($service) + }}; +} /// Maximum inbound gRPC message size (1 MB). /// @@ -57,12 +135,35 @@ impl MultiplexService { .max_decoding_message_size(MAX_GRPC_DECODE_SIZE); let inference = InferenceServer::new(InferenceService::new(self.state.clone())) .max_decoding_message_size(MAX_GRPC_DECODE_SIZE); - let grpc_service = GrpcRouter::new(openshell, inference); + let authz_policy = self.state.config.oidc.as_ref().map(|oidc| AuthzPolicy { + admin_role: oidc.admin_role.clone(), + user_role: oidc.user_role.clone(), + scopes_enabled: !oidc.scopes_claim.is_empty(), + }); + let grpc_service = AuthGrpcRouter::new( + GrpcRouter::new(openshell, inference), + self.state.oidc_cache.clone(), + authz_policy, + self.state.config.ssh_handshake_secret.clone(), + ); let http_service = http_router(self.state.clone()); + let grpc_service = request_id_middleware!(grpc_service); + let http_service = request_id_middleware!(http_service); + let service = MultiplexedService::new(grpc_service, http_service); - Builder::new(TokioExecutor::new()) + // HTTP/2 adaptive flow control. Default windows (64 KiB / 64 KiB) + // throttle the RelayStream data plane to ~500 Mbps on LAN. Instead + // of committing to a fixed large window (which worst-case pins + // `max_concurrent_streams × stream_window` bytes per connection), + // we let hyper/h2 auto-size based on the measured bandwidth-delay + // product. Idle streams stay tiny; busy bulk streams grow as + // needed. Overrides any fixed initial_*_window_size settings. + let mut builder = Builder::new(TokioExecutor::new()); + builder.http2().adaptive_window(true); + + builder .serve_connection_with_upgrades(TokioIo::new(stream), service) .await?; @@ -123,6 +224,145 @@ where } } +/// gRPC router wrapper that authenticates and authorizes requests. +/// +/// When `oidc_cache` is `Some`, extracts the `authorization: Bearer ` +/// header, validates the JWT (authentication), then checks RBAC roles +/// (authorization) before forwarding to the inner gRPC router. +/// +/// Authentication is provider-specific (currently OIDC via `oidc.rs`). +/// Authorization is provider-agnostic (via `authz.rs`). This separation +/// aligns with RFC 0001's control-plane identity design. +#[derive(Clone)] +pub struct AuthGrpcRouter { + inner: S, + oidc_cache: Option>, + authz_policy: Option, + /// SSH handshake secret used to validate sandbox-to-server RPCs. + sandbox_secret: String, +} + +impl AuthGrpcRouter { + fn new( + inner: S, + oidc_cache: Option>, + authz_policy: Option, + sandbox_secret: String, + ) -> Self { + Self { + inner, + oidc_cache, + authz_policy, + sandbox_secret, + } + } +} + +impl tower::Service> for AuthGrpcRouter +where + S: tower::Service, Response = Response> + + Clone + + Send + + 'static, + S::Future: Send, + S::Error: Send + Into>, + B: Send + 'static, +{ + type Response = S::Response; + type Error = S::Error; + type Future = Pin> + Send>>; + + fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll> { + Poll::Ready(Ok(())) + } + + fn call(&mut self, req: Request) -> Self::Future { + let oidc_cache = self.oidc_cache.clone(); + let authz_policy = self.authz_policy.clone(); + let sandbox_secret = self.sandbox_secret.clone(); + let mut inner = self.inner.clone(); + + Box::pin(async move { + let mut req = req; + oidc::clear_internal_auth_markers(req.headers_mut()); + + // If OIDC is not configured, pass through directly. + let Some(cache) = oidc_cache else { + return inner.ready().await?.call(req).await; + }; + + let path = req.uri().path().to_string(); + + // Health probes and reflection — truly unauthenticated. + if oidc::is_unauthenticated_method(&path) { + return inner.ready().await?.call(req).await; + } + + // Sandbox-to-server RPCs — authenticated via shared secret, + // not OIDC Bearer tokens. + if oidc::is_sandbox_secret_method(&path) { + if let Err(status) = oidc::validate_sandbox_secret(req.headers(), &sandbox_secret) { + let response = status.into_http(); + let (parts, body) = response.into_parts(); + let body = tonic::body::BoxBody::new(body); + return Ok(Response::from_parts(parts, body)); + } + oidc::mark_sandbox_secret_authenticated(req.headers_mut()); + return inner.ready().await?.call(req).await; + } + + // Dual-auth methods (e.g. UpdateConfig) — accept either a + // Bearer token (CLI users) or sandbox secret (supervisor). + if oidc::is_dual_auth_method(&path) + && oidc::validate_sandbox_secret(req.headers(), &sandbox_secret).is_ok() + { + oidc::mark_sandbox_secret_authenticated(req.headers_mut()); + return inner.ready().await?.call(req).await; + } + // Fall through to Bearer token validation below. + + // Extract Bearer token from the authorization header. + let token = req + .headers() + .get("authorization") + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.strip_prefix("Bearer ")); + + let Some(token) = token else { + let status = tonic::Status::unauthenticated("missing authorization header"); + let response = status.into_http(); + // Convert the response body type. + let (parts, body) = response.into_parts(); + let body = tonic::body::BoxBody::new(body); + return Ok(Response::from_parts(parts, body)); + }; + + // Authenticate: validate the JWT and produce an Identity. + let identity = match cache.validate_token(token).await { + Ok(id) => id, + Err(status) => { + let response = status.into_http(); + let (parts, body) = response.into_parts(); + let body = tonic::body::BoxBody::new(body); + return Ok(Response::from_parts(parts, body)); + } + }; + + // Authorize: check RBAC roles against the method. + if let Some(ref policy) = authz_policy + && let Err(status) = policy.check(&identity, &path) + { + let response = status.into_http(); + let (parts, body) = response.into_parts(); + let body = tonic::body::BoxBody::new(body); + return Ok(Response::from_parts(parts, body)); + } + + inner.ready().await?.call(req).await + }) + } +} + /// Service that multiplexes between gRPC and HTTP. #[derive(Clone)] pub struct MultiplexedService { @@ -162,6 +402,8 @@ where .is_some_and(|v| v.as_bytes().starts_with(b"application/grpc")); if is_grpc { + let method = grpc_method_from_path(req.uri().path()); + let start = Instant::now(); let mut grpc = self.grpc.clone(); Box::pin(async move { let (parts, body) = req.into_parts(); @@ -176,11 +418,18 @@ where .await .map_err(Into::into)?; + let code = grpc_status_from_response(&res); + let elapsed = start.elapsed().as_secs_f64(); + counter!("openshell_server_grpc_requests_total", "method" => method.clone(), "code" => code.clone()).increment(1); + histogram!("openshell_server_grpc_request_duration_seconds", "method" => method, "code" => code).record(elapsed); + let (parts, body) = res.into_parts(); let body = body.map_err(Into::into).boxed_unsync(); Ok(Response::from_parts(parts, BoxBody(body))) }) } else { + let path = normalize_http_path(req.uri().path()); + let start = Instant::now(); let mut http = self.http.clone(); Box::pin(async move { let (parts, body) = req.into_parts(); @@ -195,6 +444,11 @@ where .await .map_err(Into::into)?; + let status = res.status().as_u16().to_string(); + let elapsed = start.elapsed().as_secs_f64(); + counter!("openshell_server_http_requests_total", "path" => path, "status" => status.clone()).increment(1); + histogram!("openshell_server_http_request_duration_seconds", "path" => path, "status" => status).record(elapsed); + let (parts, body) = res.into_parts(); let body = body.map_err(Into::into).boxed_unsync(); Ok(Response::from_parts(parts, BoxBody(body))) @@ -203,6 +457,26 @@ where } } +fn grpc_method_from_path(path: &str) -> String { + path.rsplit('/').next().unwrap_or(path).to_string() +} + +fn grpc_status_from_response(res: &Response) -> String { + res.headers() + .get("grpc-status") + .and_then(|v| v.to_str().ok()) + .map_or_else(|| "0".to_string(), ToString::to_string) +} + +fn normalize_http_path(path: &str) -> &'static str { + match path { + p if p.starts_with("/connect/ssh") => "/connect/ssh", + p if p.starts_with("/_ws_tunnel") => "/_ws_tunnel", + p if p.starts_with("/auth/") => "/auth", + _ => "unknown", + } +} + /// Boxed body type for uniform handling. pub struct BoxBody( http_body_util::combinators::UnsyncBoxBody>, @@ -227,3 +501,277 @@ impl Body for BoxBody { self.0.size_hint() } } + +#[cfg(test)] +mod tests { + use super::*; + use bytes::Bytes; + use http_body_util::Empty; + use std::sync::Mutex; + + #[test] + fn uuid_request_id_generates_valid_uuid() { + let mut maker = UuidRequestId; + let req = Request::builder().body(()).unwrap(); + let id = maker.make_request_id(&req).expect("should produce an ID"); + let value = id.header_value().to_str().unwrap(); + uuid::Uuid::parse_str(value).expect("should be a valid UUID"); + } + + #[test] + fn uuid_request_id_generates_unique_ids() { + let mut maker = UuidRequestId; + let req = Request::builder().body(()).unwrap(); + let id1 = maker.make_request_id(&req).unwrap(); + let id2 = maker.make_request_id(&req).unwrap(); + assert_ne!(id1.header_value(), id2.header_value()); + } + + async fn start_http_server_with_middleware() -> std::net::SocketAddr { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + + let http_service = crate::http::health_router(); + let http_service = request_id_middleware!(http_service); + + let service = MultiplexedService::new(http_service.clone(), http_service); + + tokio::spawn(async move { + loop { + let Ok((stream, _)) = listener.accept().await else { + continue; + }; + let svc = service.clone(); + tokio::spawn(async move { + let _ = Builder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(stream), svc) + .await; + }); + } + }); + + addr + } + + async fn http1_get( + addr: std::net::SocketAddr, + path: &str, + headers: &[(&str, &str)], + ) -> Response { + let stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + let (mut sender, conn) = hyper::client::conn::http1::Builder::new() + .handshake(TokioIo::new(stream)) + .await + .unwrap(); + tokio::spawn(async move { + let _ = conn.await; + }); + + let mut builder = Request::builder() + .method("GET") + .uri(format!("http://{addr}{path}")); + for (k, v) in headers { + builder = builder.header(*k, *v); + } + let req = builder.body(Empty::::new()).unwrap(); + sender.send_request(req).await.unwrap() + } + + #[tokio::test] + async fn http_response_includes_request_id() { + let addr = start_http_server_with_middleware().await; + let resp = http1_get(addr, "/healthz", &[]).await; + assert_eq!(resp.status(), 200); + + let request_id = resp + .headers() + .get("x-request-id") + .expect("response should include x-request-id header"); + let id_str = request_id.to_str().unwrap(); + uuid::Uuid::parse_str(id_str).expect("should be a valid UUID"); + } + + #[tokio::test] + async fn http_preserves_client_request_id() { + let addr = start_http_server_with_middleware().await; + let client_id = "my-custom-correlation-id"; + let resp = http1_get(addr, "/healthz", &[("x-request-id", client_id)]).await; + assert_eq!(resp.status(), 200); + + let request_id = resp + .headers() + .get("x-request-id") + .expect("response should include x-request-id header"); + assert_eq!(request_id.to_str().unwrap(), client_id); + } + + #[tokio::test] + async fn each_request_gets_unique_id() { + let addr = start_http_server_with_middleware().await; + + let mut ids = Vec::new(); + for _ in 0..3 { + let resp = http1_get(addr, "/healthz", &[]).await; + let id = resp + .headers() + .get("x-request-id") + .unwrap() + .to_str() + .unwrap() + .to_string(); + ids.push(id); + } + + assert_ne!(ids[0], ids[1]); + assert_ne!(ids[1], ids[2]); + assert_ne!(ids[0], ids[2]); + } + + #[tokio::test] + async fn grpc_path_includes_request_id() { + let addr = start_http_server_with_middleware().await; + let resp = http1_get( + addr, + "/openshell.v1.OpenShell/Health", + &[ + ("content-type", "application/grpc"), + ("x-request-id", "grpc-corr-id"), + ], + ) + .await; + + let request_id = resp + .headers() + .get("x-request-id") + .expect("gRPC-routed response should include x-request-id header"); + assert_eq!(request_id.to_str().unwrap(), "grpc-corr-id"); + } + + #[derive(Clone)] + struct TraceBuf(Arc>>); + + impl std::io::Write for TraceBuf { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + self.0.lock().unwrap().extend_from_slice(buf); + Ok(buf.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + + #[test] + fn request_id_appears_in_trace_span() { + use tracing_subscriber::fmt::format::FmtSpan; + use tracing_subscriber::layer::SubscriberExt; + + let log_buf: Arc>> = Arc::new(Mutex::new(Vec::new())); + let writer = TraceBuf(log_buf.clone()); + + let fmt_layer = tracing_subscriber::fmt::layer() + .with_writer(move || writer.clone()) + .with_ansi(false) + .with_span_events(FmtSpan::CLOSE); + + let subscriber = tracing_subscriber::registry().with(fmt_layer); + let _guard = tracing::subscriber::set_default(subscriber); + + let req = Request::builder() + .uri("/test-path") + .header("x-request-id", "trace-test-id-12345") + .body(Empty::::new()) + .unwrap(); + let span = make_request_span(&req); + drop(span.enter()); + drop(span); + + let output = String::from_utf8(log_buf.lock().unwrap().clone()).unwrap(); + assert!( + output.contains("trace-test-id-12345"), + "trace output should contain the request_id recorded in the span, got: {output}" + ); + } + + #[test] + fn grpc_method_extracts_last_segment() { + assert_eq!( + grpc_method_from_path("/openshell.v1.OpenShell/CreateSandbox"), + "CreateSandbox" + ); + } + + #[test] + fn grpc_method_extracts_inference_service() { + assert_eq!( + grpc_method_from_path("/openshell.inference.v1.Inference/GetInferenceBundle"), + "GetInferenceBundle" + ); + } + + #[test] + fn grpc_method_handles_bare_path() { + assert_eq!(grpc_method_from_path("Health"), "Health"); + } + + #[test] + fn grpc_method_handles_single_slash() { + assert_eq!(grpc_method_from_path("/"), ""); + } + + #[test] + fn grpc_method_handles_empty_string() { + assert_eq!(grpc_method_from_path(""), ""); + } + + #[test] + fn normalize_ssh_path() { + assert_eq!(normalize_http_path("/connect/ssh"), "/connect/ssh"); + } + + #[test] + fn normalize_ssh_path_with_trailing_segments() { + assert_eq!( + normalize_http_path("/connect/ssh?token=abc"), + "/connect/ssh" + ); + } + + #[test] + fn normalize_ws_tunnel() { + assert_eq!(normalize_http_path("/_ws_tunnel"), "/_ws_tunnel"); + } + + #[test] + fn normalize_ws_tunnel_with_trailing() { + assert_eq!(normalize_http_path("/_ws_tunnel/foo"), "/_ws_tunnel"); + } + + #[test] + fn normalize_auth_path() { + assert_eq!(normalize_http_path("/auth/connect"), "/auth"); + } + + #[test] + fn normalize_auth_with_query() { + assert_eq!( + normalize_http_path("/auth/connect?callback_port=12345&code=AB7-X9KM"), + "/auth" + ); + } + + #[test] + fn normalize_unknown_path_collapses_to_unknown() { + assert_eq!(normalize_http_path("/random/scanner/probe"), "unknown"); + } + + #[test] + fn normalize_empty_path() { + assert_eq!(normalize_http_path(""), "unknown"); + } + + #[test] + fn normalize_root_path() { + assert_eq!(normalize_http_path("/"), "unknown"); + } +} diff --git a/crates/openshell-server/src/persistence/mod.rs b/crates/openshell-server/src/persistence/mod.rs index 5cd36693b9..1c926bd4a3 100644 --- a/crates/openshell-server/src/persistence/mod.rs +++ b/crates/openshell-server/src/persistence/mod.rs @@ -6,14 +6,67 @@ mod postgres; mod sqlite; -use openshell_core::{Error, Result}; +pub use openshell_core::proto::{ + StoredDraftChunk as DraftChunkRecord, StoredPolicyRevision as PolicyRecord, +}; + +use openshell_core::{Error as CoreError, Result as CoreResult}; use prost::Message; use rand::Rng; +use std::collections::HashMap; use std::time::{SystemTime, UNIX_EPOCH}; +use thiserror::Error; pub use postgres::PostgresStore; pub use sqlite::SqliteStore; +pub type PersistenceResult = Result; + +/// Persistence-layer error type. +#[derive(Debug, Error, Clone)] +pub enum PersistenceError { + #[error("configuration error: {0}")] + Config(String), + #[error("database error: {0}")] + Database(String), + #[error("migration error: {0}")] + Migration(String), + #[error("decode error: {0}")] + Decode(String), + #[error("encode error: {0}")] + Encode(String), + #[error("unique violation{constraint_msg}")] + UniqueViolation { + constraint: Option, + detail: Option, + constraint_msg: String, + }, +} + +impl PersistenceError { + pub fn unique_violation(constraint: Option, detail: Option) -> Self { + let constraint_msg = constraint + .as_ref() + .map(|value| format!(" on {value}")) + .unwrap_or_default(); + Self::UniqueViolation { + constraint, + detail, + constraint_msg, + } + } + + pub fn is_unique_violation_on(&self, constraint: &str) -> bool { + matches!( + self, + Self::UniqueViolation { + constraint: Some(value), + .. + } if value == constraint + ) + } +} + /// Stored object record. #[derive(Debug, Clone)] pub struct ObjectRecord { @@ -23,20 +76,8 @@ pub struct ObjectRecord { pub payload: Vec, pub created_at_ms: i64, pub updated_at_ms: i64, -} - -/// Stored sandbox policy revision record. -#[derive(Debug, Clone)] -pub struct PolicyRecord { - pub id: String, - pub sandbox_id: String, - pub version: i64, - pub policy_payload: Vec, - pub policy_hash: String, - pub status: String, - pub load_error: Option, - pub created_at_ms: i64, - pub loaded_at_ms: Option, + /// JSON-serialized labels (key-value pairs). + pub labels: Option, } /// Persistence store implementations. @@ -51,15 +92,9 @@ pub trait ObjectType { fn object_type() -> &'static str; } -/// Trait for extracting an object id from a message instance. -pub trait ObjectId { - fn object_id(&self) -> &str; -} - -/// Trait for extracting an object name from a message instance. -pub trait ObjectName { - fn object_name(&self) -> &str; -} +// Import object metadata accessor traits from openshell-core +// (implementations for all proto types are in openshell-core::metadata) +pub use openshell_core::{ObjectId, ObjectLabels, ObjectName}; /// Generate a random 6-character lowercase alphabetic name. pub fn generate_name() -> String { @@ -71,32 +106,53 @@ pub fn generate_name() -> String { impl Store { /// Connect to a persistence store based on the database URL. - pub async fn connect(url: &str) -> Result { + pub async fn connect(url: &str) -> CoreResult { if url.starts_with("postgres://") || url.starts_with("postgresql://") { - let store = PostgresStore::connect(url).await?; - store.migrate().await?; + let store = PostgresStore::connect(url) + .await + .map_err(|e| CoreError::execution(e.to_string()))?; + store + .migrate() + .await + .map_err(|e| CoreError::execution(e.to_string()))?; Ok(Self::Postgres(store)) } else if url.starts_with("sqlite:") { - let store = SqliteStore::connect(url).await?; - store.migrate().await?; + let store = SqliteStore::connect(url) + .await + .map_err(|e| CoreError::execution(e.to_string()))?; + store + .migrate() + .await + .map_err(|e| CoreError::execution(e.to_string()))?; Ok(Self::Sqlite(store)) } else { - Err(Error::config(format!( + Err(CoreError::config(format!( "unsupported database URL scheme: {url}" ))) } } - /// Insert or update an object. - pub async fn put(&self, object_type: &str, id: &str, name: &str, payload: &[u8]) -> Result<()> { + /// Insert or update a generic named object. + pub async fn put( + &self, + object_type: &str, + id: &str, + name: &str, + payload: &[u8], + labels: Option<&str>, + ) -> PersistenceResult<()> { match self { - Self::Postgres(store) => store.put(object_type, id, name, payload).await, - Self::Sqlite(store) => store.put(object_type, id, name, payload).await, + Self::Postgres(store) => store.put(object_type, id, name, payload, labels).await, + Self::Sqlite(store) => store.put(object_type, id, name, payload, labels).await, } } /// Fetch an object by id. - pub async fn get(&self, object_type: &str, id: &str) -> Result> { + pub async fn get( + &self, + object_type: &str, + id: &str, + ) -> PersistenceResult> { match self { Self::Postgres(store) => store.get(object_type, id).await, Self::Sqlite(store) => store.get(object_type, id).await, @@ -104,7 +160,11 @@ impl Store { } /// Fetch an object by name within an object type. - pub async fn get_by_name(&self, object_type: &str, name: &str) -> Result> { + pub async fn get_by_name( + &self, + object_type: &str, + name: &str, + ) -> PersistenceResult> { match self { Self::Postgres(store) => store.get_by_name(object_type, name).await, Self::Sqlite(store) => store.get_by_name(object_type, name).await, @@ -112,7 +172,7 @@ impl Store { } /// Delete an object by id. - pub async fn delete(&self, object_type: &str, id: &str) -> Result { + pub async fn delete(&self, object_type: &str, id: &str) -> PersistenceResult { match self { Self::Postgres(store) => store.delete(object_type, id).await, Self::Sqlite(store) => store.delete(object_type, id).await, @@ -120,7 +180,7 @@ impl Store { } /// Delete an object by name within an object type. - pub async fn delete_by_name(&self, object_type: &str, name: &str) -> Result { + pub async fn delete_by_name(&self, object_type: &str, name: &str) -> PersistenceResult { match self { Self::Postgres(store) => store.delete_by_name(object_type, name).await, Self::Sqlite(store) => store.delete_by_name(object_type, name).await, @@ -133,119 +193,31 @@ impl Store { object_type: &str, limit: u32, offset: u32, - ) -> Result> { + ) -> PersistenceResult> { match self { Self::Postgres(store) => store.list(object_type, limit, offset).await, Self::Sqlite(store) => store.list(object_type, limit, offset).await, } } - // ----------------------------------------------------------------------- - // Policy revision operations - // ----------------------------------------------------------------------- - - /// Insert a new policy revision. - pub async fn put_policy_revision( - &self, - id: &str, - sandbox_id: &str, - version: i64, - payload: &[u8], - hash: &str, - ) -> Result<()> { - match self { - Self::Postgres(store) => { - store - .put_policy_revision(id, sandbox_id, version, payload, hash) - .await - } - Self::Sqlite(store) => { - store - .put_policy_revision(id, sandbox_id, version, payload, hash) - .await - } - } - } - - /// Get the latest policy revision for a sandbox (by highest version, any status). - pub async fn get_latest_policy(&self, sandbox_id: &str) -> Result> { - match self { - Self::Postgres(store) => store.get_latest_policy(sandbox_id).await, - Self::Sqlite(store) => store.get_latest_policy(sandbox_id).await, - } - } - - /// Get the latest loaded policy revision for a sandbox. - pub async fn get_latest_loaded_policy(&self, sandbox_id: &str) -> Result> { - match self { - Self::Postgres(store) => store.get_latest_loaded_policy(sandbox_id).await, - Self::Sqlite(store) => store.get_latest_loaded_policy(sandbox_id).await, - } - } - - /// Get a specific policy revision by sandbox id and version. - pub async fn get_policy_by_version( - &self, - sandbox_id: &str, - version: i64, - ) -> Result> { - match self { - Self::Postgres(store) => store.get_policy_by_version(sandbox_id, version).await, - Self::Sqlite(store) => store.get_policy_by_version(sandbox_id, version).await, - } - } - - /// List policy revisions for a sandbox, ordered by version descending. - pub async fn list_policies( + /// List objects by type with label selector filtering. + /// Label selector format: "key1=value1,key2=value2" (comma-separated equality matches). + pub async fn list_with_selector( &self, - sandbox_id: &str, + object_type: &str, + label_selector: &str, limit: u32, offset: u32, - ) -> Result> { - match self { - Self::Postgres(store) => store.list_policies(sandbox_id, limit, offset).await, - Self::Sqlite(store) => store.list_policies(sandbox_id, limit, offset).await, - } - } - - /// Update the status of a policy revision. - pub async fn update_policy_status( - &self, - sandbox_id: &str, - version: i64, - status: &str, - load_error: Option<&str>, - loaded_at_ms: Option, - ) -> Result { + ) -> PersistenceResult> { match self { Self::Postgres(store) => { store - .update_policy_status(sandbox_id, version, status, load_error, loaded_at_ms) + .list_with_selector(object_type, label_selector, limit, offset) .await } Self::Sqlite(store) => { store - .update_policy_status(sandbox_id, version, status, load_error, loaded_at_ms) - .await - } - } - } - - /// Mark all pending and loaded policy revisions older than `before_version` as superseded. - pub async fn supersede_older_policies( - &self, - sandbox_id: &str, - before_version: i64, - ) -> Result { - match self { - Self::Postgres(store) => { - store - .supersede_older_policies(sandbox_id, before_version) - .await - } - Self::Sqlite(store) => { - store - .supersede_older_policies(sandbox_id, before_version) + .list_with_selector(object_type, label_selector, limit, offset) .await } } @@ -256,15 +228,26 @@ impl Store { // ----------------------------------------------------------------------- /// Insert or update a protobuf message using its inferred object type, id, and name. - pub async fn put_message( + pub async fn put_message( &self, message: &T, - ) -> Result<()> { + ) -> PersistenceResult<()> { + // Serialize labels to JSON + let labels_map = message.object_labels(); + let labels_json = if labels_map.as_ref().is_none_or(HashMap::is_empty) { + None + } else { + Some(serde_json::to_string(&labels_map).map_err(|e| { + PersistenceError::Encode(format!("failed to serialize labels: {e}")) + })?) + }; + self.put( T::object_type(), message.object_id(), message.object_name(), &message.encode_to_vec(), + labels_json.as_deref(), ) .await } @@ -273,7 +256,7 @@ impl Store { pub async fn get_message( &self, id: &str, - ) -> Result> { + ) -> PersistenceResult> { let record = self.get(T::object_type(), id).await?; let Some(record) = record else { return Ok(None); @@ -281,14 +264,14 @@ impl Store { T::decode(record.payload.as_slice()) .map(Some) - .map_err(|e| Error::execution(format!("protobuf decode error: {e}"))) + .map_err(|e| PersistenceError::Decode(format!("protobuf decode error: {e}"))) } /// Fetch and decode a protobuf message by name. pub async fn get_message_by_name( &self, name: &str, - ) -> Result> { + ) -> PersistenceResult> { let record = self.get_by_name(T::object_type(), name).await?; let Some(record) = record else { return Ok(None); @@ -296,129 +279,88 @@ impl Store { T::decode(record.payload.as_slice()) .map(Some) - .map_err(|e| Error::execution(format!("protobuf decode error: {e}"))) + .map_err(|e| PersistenceError::Decode(format!("protobuf decode error: {e}"))) } +} - // ----------------------------------------------------------------------- - // Draft policy chunk operations - // ----------------------------------------------------------------------- - - /// Insert a new draft policy chunk. - pub async fn put_draft_chunk(&self, chunk: &DraftChunkRecord) -> Result<()> { - match self { - Self::Postgres(store) => store.put_draft_chunk(chunk).await, - Self::Sqlite(store) => store.put_draft_chunk(chunk).await, - } - } +pub fn current_time_ms() -> PersistenceResult { + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|e| PersistenceError::Database(format!("time error: {e}")))?; + i64::try_from(now.as_millis()) + .map_err(|e| PersistenceError::Database(format!("time conversion error: {e}"))) +} - /// Fetch a single draft chunk by id. - pub async fn get_draft_chunk(&self, id: &str) -> Result> { - match self { - Self::Postgres(store) => store.get_draft_chunk(id).await, - Self::Sqlite(store) => store.get_draft_chunk(id).await, - } +fn map_db_error(error: &sqlx::Error) -> PersistenceError { + if let sqlx::Error::Database(db) = error + && db.is_unique_violation() + { + let constraint = db + .constraint() + .map(ToString::to_string) + .or_else(|| infer_sqlite_unique_constraint(db.message())); + return PersistenceError::unique_violation(constraint, Some(db.message().to_string())); } + PersistenceError::Database(error.to_string()) +} - /// List draft chunks for a sandbox, optionally filtered by status. - pub async fn list_draft_chunks( - &self, - sandbox_id: &str, - status_filter: Option<&str>, - ) -> Result> { - match self { - Self::Postgres(store) => store.list_draft_chunks(sandbox_id, status_filter).await, - Self::Sqlite(store) => store.list_draft_chunks(sandbox_id, status_filter).await, - } +fn infer_sqlite_unique_constraint(message: &str) -> Option { + if message.contains("objects.object_type, objects.scope, objects.version") { + Some("objects_version_uq".to_string()) + } else if message.contains("objects.object_type, objects.scope, objects.dedup_key") { + Some("objects_dedup_uq".to_string()) + } else if message.contains("objects.object_type, objects.name") { + Some("objects_name_uq".to_string()) + } else if message.contains("objects.id") { + Some("objects_pkey".to_string()) + } else { + None } +} - /// Update the status of a draft chunk. - pub async fn update_draft_chunk_status( - &self, - id: &str, - status: &str, - decided_at_ms: Option, - ) -> Result { - match self { - Self::Postgres(store) => { - store - .update_draft_chunk_status(id, status, decided_at_ms) - .await - } - Self::Sqlite(store) => { - store - .update_draft_chunk_status(id, status, decided_at_ms) - .await - } - } - } +fn map_migrate_error(error: &sqlx::migrate::MigrateError) -> PersistenceError { + PersistenceError::Migration(error.to_string()) +} - /// Update the proposed rule on a pending draft chunk. - pub async fn update_draft_chunk_rule(&self, id: &str, proposed_rule: &[u8]) -> Result { - match self { - Self::Postgres(store) => store.update_draft_chunk_rule(id, proposed_rule).await, - Self::Sqlite(store) => store.update_draft_chunk_rule(id, proposed_rule).await, - } +/// Parse a simple label selector string into key-value pairs. +/// Format: "key1=value1,key2=value2" +/// Returns a `HashMap` of label requirements. +/// +/// Note: Input validation should be performed at the gRPC layer using +/// `grpc::validation::validate_label_selector()` before calling this function. +/// Errors returned here indicate unexpected internal errors, not user input errors. +pub fn parse_label_selector(selector: &str) -> PersistenceResult> { + if selector.is_empty() { + return Ok(HashMap::new()); } - /// Delete all draft chunks for a sandbox with a given status. - pub async fn delete_draft_chunks(&self, sandbox_id: &str, status: &str) -> Result { - match self { - Self::Postgres(store) => store.delete_draft_chunks(sandbox_id, status).await, - Self::Sqlite(store) => store.delete_draft_chunks(sandbox_id, status).await, + let mut labels = HashMap::new(); + for pair in selector.split(',') { + let pair = pair.trim(); + if pair.is_empty() { + continue; } - } - /// Get the current maximum draft version for a sandbox. - pub async fn get_draft_version(&self, sandbox_id: &str) -> Result { - match self { - Self::Postgres(store) => store.get_draft_version(sandbox_id).await, - Self::Sqlite(store) => store.get_draft_version(sandbox_id).await, + let parts: Vec<&str> = pair.splitn(2, '=').collect(); + if parts.len() != 2 { + return Err(PersistenceError::Decode(format!( + "invalid label selector: expected 'key=value', got '{pair}'" + ))); } - } -} -/// Stored draft policy chunk record. -#[derive(Debug, Clone)] -pub struct DraftChunkRecord { - pub id: String, - pub sandbox_id: String, - pub draft_version: i64, - pub status: String, - pub rule_name: String, - pub proposed_rule: Vec, - pub rationale: String, - pub security_notes: String, - pub confidence: f64, - pub created_at_ms: i64, - pub decided_at_ms: Option, - /// Denormalized endpoint host (lowercase) for DB-level dedup. - pub host: String, - /// Denormalized endpoint port for DB-level dedup. - pub port: i32, - /// Binary path that triggered the denial (for per-binary dedup). - pub binary: String, - /// How many times this endpoint has been seen across denial flush cycles. - pub hit_count: i32, - /// First time this endpoint was proposed (ms since epoch). - pub first_seen_ms: i64, - /// Most recent time this endpoint was re-proposed (ms since epoch). - pub last_seen_ms: i64, -} + let key = parts[0].trim(); + let value = parts[1].trim(); -fn current_time_ms() -> Result { - let now = SystemTime::now() - .duration_since(UNIX_EPOCH) - .map_err(|e| Error::execution(format!("time error: {e}")))?; - i64::try_from(now.as_millis()) - .map_err(|e| Error::execution(format!("time conversion error: {e}"))) -} + if key.is_empty() { + return Err(PersistenceError::Decode(format!( + "invalid label selector: key cannot be empty in '{pair}'" + ))); + } -fn map_db_error(error: &sqlx::Error) -> Error { - Error::execution(format!("database error: {error}")) -} + labels.insert(key.to_string(), value.to_string()); + } -fn map_migrate_error(error: &sqlx::migrate::MigrateError) -> Error { - Error::execution(format!("migration error: {error}")) + Ok(labels) } #[cfg(test)] diff --git a/crates/openshell-server/src/persistence/postgres.rs b/crates/openshell-server/src/persistence/postgres.rs index 4b62516c4d..2cd6a046f1 100644 --- a/crates/openshell-server/src/persistence/postgres.rs +++ b/crates/openshell-server/src/persistence/postgres.rs @@ -2,12 +2,20 @@ // SPDX-License-Identifier: Apache-2.0 use super::{ - DraftChunkRecord, ObjectRecord, PolicyRecord, current_time_ms, map_db_error, map_migrate_error, + DraftChunkRecord, ObjectRecord, PersistenceResult, PolicyRecord, current_time_ms, map_db_error, + map_migrate_error, +}; +use crate::policy_store::{ + draft_chunk_payload_from_record, draft_chunk_record_from_parts, policy_payload_from_record, + policy_record_from_parts, }; -use openshell_core::Result; use sqlx::postgres::PgPoolOptions; use sqlx::{PgPool, Row}; -use std::path::PathBuf; + +static POSTGRES_MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations/postgres"); + +const POLICY_OBJECT_TYPE: &str = "sandbox_policy"; +const DRAFT_CHUNK_OBJECT_TYPE: &str = "draft_policy_chunk"; #[derive(Debug, Clone)] pub struct PostgresStore { @@ -15,7 +23,7 @@ pub struct PostgresStore { } impl PostgresStore { - pub async fn connect(url: &str) -> Result { + pub async fn connect(url: &str) -> PersistenceResult { let pool = PgPoolOptions::new() .max_connections(10) .connect(url) @@ -25,29 +33,35 @@ impl PostgresStore { Ok(Self { pool }) } - pub async fn migrate(&self) -> Result<()> { - let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .join("migrations") - .join("postgres"); - let migrator = sqlx::migrate::Migrator::new(path) - .await - .map_err(|e| map_migrate_error(&e))?; - migrator + pub async fn migrate(&self) -> PersistenceResult<()> { + POSTGRES_MIGRATOR .run(&self.pool) .await .map_err(|e| map_migrate_error(&e)) } - pub async fn put(&self, object_type: &str, id: &str, name: &str, payload: &[u8]) -> Result<()> { + pub async fn put( + &self, + object_type: &str, + id: &str, + name: &str, + payload: &[u8], + labels: Option<&str>, + ) -> PersistenceResult<()> { let now_ms = current_time_ms()?; + let labels_jsonb: Option = labels + .map(serde_json::from_str) + .transpose() + .map_err(|e| super::PersistenceError::Encode(format!("invalid labels JSON: {e}")))?; + sqlx::query( r" -INSERT INTO objects (object_type, id, name, payload, created_at_ms, updated_at_ms) -VALUES ($1, $2, $3, $4, $5, $5) -ON CONFLICT (id) DO UPDATE SET +INSERT INTO objects (object_type, id, name, payload, created_at_ms, updated_at_ms, labels) +VALUES ($1, $2, $3, $4, $5, $5, COALESCE($6, '{}'::jsonb)) +ON CONFLICT (object_type, name) WHERE name IS NOT NULL DO UPDATE SET payload = EXCLUDED.payload, - updated_at_ms = EXCLUDED.updated_at_ms -WHERE objects.object_type = EXCLUDED.object_type + updated_at_ms = EXCLUDED.updated_at_ms, + labels = EXCLUDED.labels ", ) .bind(object_type) @@ -55,16 +69,21 @@ WHERE objects.object_type = EXCLUDED.object_type .bind(name) .bind(payload) .bind(now_ms) + .bind(labels_jsonb) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; Ok(()) } - pub async fn get(&self, object_type: &str, id: &str) -> Result> { + pub async fn get( + &self, + object_type: &str, + id: &str, + ) -> PersistenceResult> { let row = sqlx::query( r" -SELECT object_type, id, name, payload, created_at_ms, updated_at_ms +SELECT object_type, id, name, payload, created_at_ms, updated_at_ms, labels FROM objects WHERE object_type = $1 AND id = $2 ", @@ -75,20 +94,17 @@ WHERE object_type = $1 AND id = $2 .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(|row| ObjectRecord { - object_type: row.get("object_type"), - id: row.get("id"), - name: row.get("name"), - payload: row.get("payload"), - created_at_ms: row.get("created_at_ms"), - updated_at_ms: row.get("updated_at_ms"), - })) + Ok(row.map(row_to_object_record)) } - pub async fn get_by_name(&self, object_type: &str, name: &str) -> Result> { + pub async fn get_by_name( + &self, + object_type: &str, + name: &str, + ) -> PersistenceResult> { let row = sqlx::query( r" -SELECT object_type, id, name, payload, created_at_ms, updated_at_ms +SELECT object_type, id, name, payload, created_at_ms, updated_at_ms, labels FROM objects WHERE object_type = $1 AND name = $2 ", @@ -99,17 +115,10 @@ WHERE object_type = $1 AND name = $2 .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(|row| ObjectRecord { - object_type: row.get("object_type"), - id: row.get("id"), - name: row.get("name"), - payload: row.get("payload"), - created_at_ms: row.get("created_at_ms"), - updated_at_ms: row.get("updated_at_ms"), - })) + Ok(row.map(row_to_object_record)) } - pub async fn delete(&self, object_type: &str, id: &str) -> Result { + pub async fn delete(&self, object_type: &str, id: &str) -> PersistenceResult { let result = sqlx::query("DELETE FROM objects WHERE object_type = $1 AND id = $2") .bind(object_type) .bind(id) @@ -119,7 +128,7 @@ WHERE object_type = $1 AND name = $2 Ok(result.rows_affected() > 0) } - pub async fn delete_by_name(&self, object_type: &str, name: &str) -> Result { + pub async fn delete_by_name(&self, object_type: &str, name: &str) -> PersistenceResult { let result = sqlx::query("DELETE FROM objects WHERE object_type = $1 AND name = $2") .bind(object_type) .bind(name) @@ -134,10 +143,10 @@ WHERE object_type = $1 AND name = $2 object_type: &str, limit: u32, offset: u32, - ) -> Result> { + ) -> PersistenceResult> { let rows = sqlx::query( r" -SELECT object_type, id, name, payload, created_at_ms, updated_at_ms +SELECT object_type, id, name, payload, created_at_ms, updated_at_ms, labels FROM objects WHERE object_type = $1 ORDER BY created_at_ms ASC, name ASC @@ -151,24 +160,42 @@ LIMIT $2 OFFSET $3 .await .map_err(|e| map_db_error(&e))?; - let records = rows - .into_iter() - .map(|row| ObjectRecord { - object_type: row.get("object_type"), - id: row.get("id"), - name: row.get("name"), - payload: row.get("payload"), - created_at_ms: row.get("created_at_ms"), - updated_at_ms: row.get("updated_at_ms"), - }) - .collect(); - - Ok(records) + Ok(rows.into_iter().map(row_to_object_record).collect()) } - // ------------------------------------------------------------------- - // Policy revision operations - // ------------------------------------------------------------------- + pub async fn list_with_selector( + &self, + object_type: &str, + label_selector: &str, + limit: u32, + offset: u32, + ) -> PersistenceResult> { + use super::parse_label_selector; + + let required_labels = parse_label_selector(label_selector)?; + let labels_jsonb = serde_json::to_value(&required_labels).map_err(|e| { + super::PersistenceError::Encode(format!("failed to serialize labels: {e}")) + })?; + + let rows = sqlx::query( + r" +SELECT object_type, id, name, payload, created_at_ms, updated_at_ms, labels +FROM objects +WHERE object_type = $1 AND labels @> $2 +ORDER BY created_at_ms ASC, name ASC +LIMIT $3 OFFSET $4 +", + ) + .bind(object_type) + .bind(&labels_jsonb) + .bind(i64::from(limit)) + .bind(i64::from(offset)) + .fetch_all(&self.pool) + .await + .map_err(|e| map_db_error(&e))?; + + Ok(rows.into_iter().map(row_to_object_record).collect()) + } pub async fn put_policy_revision( &self, @@ -177,19 +204,35 @@ LIMIT $2 OFFSET $3 version: i64, payload: &[u8], hash: &str, - ) -> Result<()> { + ) -> PersistenceResult<()> { let now_ms = current_time_ms()?; + let record = PolicyRecord { + id: id.to_string(), + sandbox_id: sandbox_id.to_string(), + version, + policy_payload: payload.to_vec(), + policy_hash: hash.to_string(), + status: "pending".to_string(), + load_error: None, + created_at_ms: now_ms, + loaded_at_ms: None, + }; + let wrapped_payload = policy_payload_from_record(&record)?; + sqlx::query( r" -INSERT INTO sandbox_policies (id, sandbox_id, version, policy_payload, policy_hash, status, created_at_ms) -VALUES ($1, $2, $3, $4, $5, 'pending', $6) +INSERT INTO objects ( + object_type, id, scope, version, status, payload, created_at_ms, updated_at_ms +) +VALUES ($1, $2, $3, $4, $5, $6, $7, $7) ", ) + .bind(POLICY_OBJECT_TYPE) .bind(id) .bind(sandbox_id) .bind(version) - .bind(payload) - .bind(hash) + .bind("pending") + .bind(wrapped_payload) .bind(now_ms) .execute(&self.pool) .await @@ -197,61 +240,70 @@ VALUES ($1, $2, $3, $4, $5, 'pending', $6) Ok(()) } - pub async fn get_latest_policy(&self, sandbox_id: &str) -> Result> { + pub async fn get_latest_policy( + &self, + sandbox_id: &str, + ) -> PersistenceResult> { let row = sqlx::query( r" -SELECT id, sandbox_id, version, policy_payload, policy_hash, status, load_error, created_at_ms, loaded_at_ms -FROM sandbox_policies -WHERE sandbox_id = $1 -ORDER BY version DESC +SELECT id, scope, version, status, payload, created_at_ms +FROM objects +WHERE object_type = $1 AND scope = $2 +ORDER BY version DESC, created_at_ms DESC LIMIT 1 ", ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .fetch_optional(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(row_to_policy_record)) + row.map(row_to_policy_record).transpose() } - pub async fn get_latest_loaded_policy(&self, sandbox_id: &str) -> Result> { + pub async fn get_latest_loaded_policy( + &self, + sandbox_id: &str, + ) -> PersistenceResult> { let row = sqlx::query( r" -SELECT id, sandbox_id, version, policy_payload, policy_hash, status, load_error, created_at_ms, loaded_at_ms -FROM sandbox_policies -WHERE sandbox_id = $1 AND status = 'loaded' -ORDER BY version DESC +SELECT id, scope, version, status, payload, created_at_ms +FROM objects +WHERE object_type = $1 AND scope = $2 AND status = 'loaded' +ORDER BY version DESC, created_at_ms DESC LIMIT 1 ", ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .fetch_optional(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(row_to_policy_record)) + row.map(row_to_policy_record).transpose() } pub async fn get_policy_by_version( &self, sandbox_id: &str, version: i64, - ) -> Result> { + ) -> PersistenceResult> { let row = sqlx::query( r" -SELECT id, sandbox_id, version, policy_payload, policy_hash, status, load_error, created_at_ms, loaded_at_ms -FROM sandbox_policies -WHERE sandbox_id = $1 AND version = $2 +SELECT id, scope, version, status, payload, created_at_ms +FROM objects +WHERE object_type = $1 AND scope = $2 AND version = $3 ", ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .bind(version) .fetch_optional(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(row_to_policy_record)) + row.map(row_to_policy_record).transpose() } pub async fn list_policies( @@ -259,16 +311,17 @@ WHERE sandbox_id = $1 AND version = $2 sandbox_id: &str, limit: u32, offset: u32, - ) -> Result> { + ) -> PersistenceResult> { let rows = sqlx::query( r" -SELECT id, sandbox_id, version, policy_payload, policy_hash, status, load_error, created_at_ms, loaded_at_ms -FROM sandbox_policies -WHERE sandbox_id = $1 -ORDER BY version DESC -LIMIT $2 OFFSET $3 +SELECT id, scope, version, status, payload, created_at_ms +FROM objects +WHERE object_type = $1 AND scope = $2 +ORDER BY version DESC, created_at_ms DESC +LIMIT $3 OFFSET $4 ", ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .bind(i64::from(limit)) .bind(i64::from(offset)) @@ -276,7 +329,7 @@ LIMIT $2 OFFSET $3 .await .map_err(|e| map_db_error(&e))?; - Ok(rows.into_iter().map(row_to_policy_record).collect()) + rows.into_iter().map(row_to_policy_record).collect() } pub async fn update_policy_status( @@ -286,19 +339,30 @@ LIMIT $2 OFFSET $3 status: &str, load_error: Option<&str>, loaded_at_ms: Option, - ) -> Result { + ) -> PersistenceResult { + let Some(mut record) = self.get_policy_by_version(sandbox_id, version).await? else { + return Ok(false); + }; + + record.status = status.to_string(); + record.load_error = load_error.map(ToOwned::to_owned); + record.loaded_at_ms = loaded_at_ms; + let payload = policy_payload_from_record(&record)?; + let now_ms = current_time_ms()?; + let result = sqlx::query( r" -UPDATE sandbox_policies -SET status = $3, load_error = $4, loaded_at_ms = $5 -WHERE sandbox_id = $1 AND version = $2 +UPDATE objects +SET status = $4, payload = $5, updated_at_ms = $6 +WHERE object_type = $1 AND scope = $2 AND version = $3 ", ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .bind(version) .bind(status) - .bind(load_error) - .bind(loaded_at_ms) + .bind(payload) + .bind(now_ms) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; @@ -309,57 +373,48 @@ WHERE sandbox_id = $1 AND version = $2 &self, sandbox_id: &str, before_version: i64, - ) -> Result { + ) -> PersistenceResult { + let now_ms = current_time_ms()?; let result = sqlx::query( r" -UPDATE sandbox_policies -SET status = 'superseded' -WHERE sandbox_id = $1 AND version < $2 AND status IN ('pending', 'loaded') +UPDATE objects +SET status = 'superseded', updated_at_ms = $4 +WHERE object_type = $1 + AND scope = $2 + AND version < $3 + AND status IN ('pending', 'loaded') ", ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .bind(before_version) + .bind(now_ms) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; Ok(result.rows_affected()) } - // ------------------------------------------------------------------- - // Draft policy chunk operations - // ------------------------------------------------------------------- - - pub async fn put_draft_chunk(&self, chunk: &DraftChunkRecord) -> Result<()> { + pub async fn put_draft_chunk(&self, chunk: &DraftChunkRecord) -> PersistenceResult<()> { + let payload = draft_chunk_payload_from_record(chunk)?; sqlx::query( r" -INSERT INTO draft_policy_chunks - (id, sandbox_id, draft_version, status, rule_name, - proposed_rule, rationale, security_notes, confidence, - created_at_ms, decided_at_ms, host, port, binary, - hit_count, first_seen_ms, last_seen_ms) -VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17) -ON CONFLICT (sandbox_id, host, port, binary) - WHERE status IN ('pending', 'approved', 'rejected') -DO UPDATE SET - hit_count = draft_policy_chunks.hit_count + EXCLUDED.hit_count, - last_seen_ms = EXCLUDED.last_seen_ms +INSERT INTO objects ( + object_type, id, scope, status, dedup_key, hit_count, payload, created_at_ms, updated_at_ms +) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) +ON CONFLICT (object_type, scope, dedup_key) WHERE dedup_key IS NOT NULL DO UPDATE SET + hit_count = objects.hit_count + EXCLUDED.hit_count, + updated_at_ms = EXCLUDED.updated_at_ms ", ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(&chunk.id) .bind(&chunk.sandbox_id) - .bind(chunk.draft_version) .bind(&chunk.status) - .bind(&chunk.rule_name) - .bind(&chunk.proposed_rule) - .bind(&chunk.rationale) - .bind(&chunk.security_notes) - .bind(chunk.confidence) - .bind(chunk.created_at_ms) - .bind(chunk.decided_at_ms) - .bind(&chunk.host) - .bind(chunk.port) - .bind(&chunk.binary) - .bind(chunk.hit_count) + .bind(draft_chunk_dedup_key(chunk)) + .bind(i64::from(chunk.hit_count)) + .bind(payload) .bind(chunk.first_seen_ms) .bind(chunk.last_seen_ms) .execute(&self.pool) @@ -368,33 +423,38 @@ DO UPDATE SET Ok(()) } - pub async fn get_draft_chunk(&self, id: &str) -> Result> { + pub async fn get_draft_chunk(&self, id: &str) -> PersistenceResult> { let row = sqlx::query( r" -SELECT * FROM draft_policy_chunks WHERE id = $1 +SELECT id, scope, status, hit_count, payload, created_at_ms, updated_at_ms +FROM objects +WHERE object_type = $1 AND id = $2 ", ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(id) .fetch_optional(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(row_to_draft_chunk_record)) + row.map(row_to_draft_chunk_record).transpose() } pub async fn list_draft_chunks( &self, sandbox_id: &str, status_filter: Option<&str>, - ) -> Result> { + ) -> PersistenceResult> { let rows = if let Some(status) = status_filter { sqlx::query( r" -SELECT * FROM draft_policy_chunks -WHERE sandbox_id = $1 AND status = $2 +SELECT id, scope, status, hit_count, payload, created_at_ms, updated_at_ms +FROM objects +WHERE object_type = $1 AND scope = $2 AND status = $3 ORDER BY created_at_ms DESC ", ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(sandbox_id) .bind(status) .fetch_all(&self.pool) @@ -402,18 +462,20 @@ ORDER BY created_at_ms DESC } else { sqlx::query( r" -SELECT * FROM draft_policy_chunks -WHERE sandbox_id = $1 +SELECT id, scope, status, hit_count, payload, created_at_ms, updated_at_ms +FROM objects +WHERE object_type = $1 AND scope = $2 ORDER BY created_at_ms DESC ", ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(sandbox_id) .fetch_all(&self.pool) .await } .map_err(|e| map_db_error(&e))?; - Ok(rows.into_iter().map(row_to_draft_chunk_record).collect()) + rows.into_iter().map(row_to_draft_chunk_record).collect() } pub async fn update_draft_chunk_status( @@ -421,46 +483,80 @@ ORDER BY created_at_ms DESC id: &str, status: &str, decided_at_ms: Option, - ) -> Result { + ) -> PersistenceResult { + let Some(mut record) = self.get_draft_chunk(id).await? else { + return Ok(false); + }; + + record.status = status.to_string(); + record.decided_at_ms = decided_at_ms; + record.last_seen_ms = current_time_ms()?; + let payload = draft_chunk_payload_from_record(&record)?; + let result = sqlx::query( r" -UPDATE draft_policy_chunks -SET status = $2, decided_at_ms = $3 -WHERE id = $1 +UPDATE objects +SET status = $3, payload = $4, updated_at_ms = $5 +WHERE object_type = $1 AND id = $2 ", ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(id) .bind(status) - .bind(decided_at_ms) + .bind(payload) + .bind(record.last_seen_ms) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; Ok(result.rows_affected() > 0) } - pub async fn update_draft_chunk_rule(&self, id: &str, proposed_rule: &[u8]) -> Result { + pub async fn update_draft_chunk_rule( + &self, + id: &str, + proposed_rule: &[u8], + ) -> PersistenceResult { + let Some(mut record) = self.get_draft_chunk(id).await? else { + return Ok(false); + }; + + if record.status != "pending" { + return Ok(false); + } + + record.proposed_rule = proposed_rule.to_vec(); + record.last_seen_ms = current_time_ms()?; + let payload = draft_chunk_payload_from_record(&record)?; + let result = sqlx::query( r" -UPDATE draft_policy_chunks -SET proposed_rule = $2 -WHERE id = $1 AND status = 'pending' +UPDATE objects +SET payload = $3, updated_at_ms = $4 +WHERE object_type = $1 AND id = $2 AND status = 'pending' ", ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(id) - .bind(proposed_rule) + .bind(payload) + .bind(record.last_seen_ms) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; Ok(result.rows_affected() > 0) } - pub async fn delete_draft_chunks(&self, sandbox_id: &str, status: &str) -> Result { + pub async fn delete_draft_chunks( + &self, + sandbox_id: &str, + status: &str, + ) -> PersistenceResult { let result = sqlx::query( r" -DELETE FROM draft_policy_chunks -WHERE sandbox_id = $1 AND status = $2 +DELETE FROM objects +WHERE object_type = $1 AND scope = $2 AND status = $3 ", ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(sandbox_id) .bind(status) .execute(&self.pool) @@ -469,55 +565,80 @@ WHERE sandbox_id = $1 AND status = $2 Ok(result.rows_affected()) } - pub async fn get_draft_version(&self, sandbox_id: &str) -> Result { - let row = sqlx::query( + pub async fn get_draft_version(&self, sandbox_id: &str) -> PersistenceResult { + let rows = sqlx::query( r" -SELECT COALESCE(MAX(draft_version), 0) as max_version -FROM draft_policy_chunks -WHERE sandbox_id = $1 +SELECT payload +FROM objects +WHERE object_type = $1 AND scope = $2 ", ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(sandbox_id) - .fetch_one(&self.pool) + .fetch_all(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.get("max_version")) + let mut max_version = 0_i64; + for row in rows { + let payload: Vec = row.get("payload"); + let wrapper = draft_chunk_record_from_parts( + String::new(), + sandbox_id.to_string(), + String::new(), + 0, + &payload, + 0, + 0, + )?; + max_version = max_version.max(wrapper.draft_version); + } + Ok(max_version) } } -fn row_to_draft_chunk_record(row: sqlx::postgres::PgRow) -> DraftChunkRecord { - DraftChunkRecord { - id: row.get("id"), - sandbox_id: row.get("sandbox_id"), - draft_version: row.get("draft_version"), - status: row.get("status"), - rule_name: row.get("rule_name"), - proposed_rule: row.get("proposed_rule"), - rationale: row.get("rationale"), - security_notes: row.get("security_notes"), - confidence: row.get("confidence"), - created_at_ms: row.get("created_at_ms"), - decided_at_ms: row.get("decided_at_ms"), - host: row.get("host"), - port: row.get("port"), - binary: row.get("binary"), - hit_count: row.get("hit_count"), - first_seen_ms: row.get("first_seen_ms"), - last_seen_ms: row.get("last_seen_ms"), - } +fn draft_chunk_dedup_key(chunk: &DraftChunkRecord) -> String { + format!("{}|{}|{}", chunk.host, chunk.port, chunk.binary) } -fn row_to_policy_record(row: sqlx::postgres::PgRow) -> PolicyRecord { - PolicyRecord { +fn row_to_object_record(row: sqlx::postgres::PgRow) -> ObjectRecord { + let labels_jsonb: Option = row.get("labels"); + ObjectRecord { + object_type: row.get("object_type"), id: row.get("id"), - sandbox_id: row.get("sandbox_id"), - version: row.get("version"), - policy_payload: row.get("policy_payload"), - policy_hash: row.get("policy_hash"), - status: row.get("status"), - load_error: row.get("load_error"), + name: row.get("name"), + payload: row.get("payload"), created_at_ms: row.get("created_at_ms"), - loaded_at_ms: row.get("loaded_at_ms"), + updated_at_ms: row.get("updated_at_ms"), + labels: labels_jsonb.map(|value| value.to_string()), } } + +fn row_to_policy_record(row: sqlx::postgres::PgRow) -> PersistenceResult { + let id: String = row.get("id"); + let sandbox_id: String = row.get("scope"); + let version: i64 = row.get("version"); + let status: String = row.get("status"); + let payload: Vec = row.get("payload"); + let created_at_ms: i64 = row.get("created_at_ms"); + policy_record_from_parts(id, sandbox_id, version, status, &payload, created_at_ms) +} + +fn row_to_draft_chunk_record(row: sqlx::postgres::PgRow) -> PersistenceResult { + let id: String = row.get("id"); + let sandbox_id: String = row.get("scope"); + let status: String = row.get("status"); + let hit_count: i64 = row.get("hit_count"); + let payload: Vec = row.get("payload"); + let created_at_ms: i64 = row.get("created_at_ms"); + let updated_at_ms: i64 = row.get("updated_at_ms"); + draft_chunk_record_from_parts( + id, + sandbox_id, + status, + hit_count, + &payload, + created_at_ms, + updated_at_ms, + ) +} diff --git a/crates/openshell-server/src/persistence/sqlite.rs b/crates/openshell-server/src/persistence/sqlite.rs index 167f3521db..fafb075971 100644 --- a/crates/openshell-server/src/persistence/sqlite.rs +++ b/crates/openshell-server/src/persistence/sqlite.rs @@ -2,34 +2,45 @@ // SPDX-License-Identifier: Apache-2.0 use super::{ - DraftChunkRecord, ObjectRecord, PolicyRecord, current_time_ms, map_db_error, map_migrate_error, + DraftChunkRecord, ObjectRecord, PersistenceResult, PolicyRecord, current_time_ms, map_db_error, + map_migrate_error, +}; +use crate::policy_store::{ + draft_chunk_payload_from_record, draft_chunk_record_from_parts, policy_payload_from_record, + policy_record_from_parts, }; -use openshell_core::Result; use sqlx::sqlite::{SqliteConnectOptions, SqlitePoolOptions}; use sqlx::{Row, SqlitePool}; -use std::path::PathBuf; use std::str::FromStr; +static SQLITE_MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations/sqlite"); + +const POLICY_OBJECT_TYPE: &str = "sandbox_policy"; +const DRAFT_CHUNK_OBJECT_TYPE: &str = "draft_policy_chunk"; + #[derive(Debug, Clone)] pub struct SqliteStore { pool: SqlitePool, } impl SqliteStore { - pub async fn connect(url: &str) -> Result { - let max_connections = if url.contains(":memory:") || url.contains("mode=memory") { - 1 - } else { - 5 - }; + pub async fn connect(url: &str) -> PersistenceResult { + let is_in_memory = url.contains(":memory:") || url.contains("mode=memory"); + let max_connections = if is_in_memory { 1 } else { 5 }; let options = SqliteConnectOptions::from_str(url) .map_err(|e| map_db_error(&e))? .create_if_missing(true); - let pool = SqlitePoolOptions::new() + let mut pool_options = SqlitePoolOptions::new() .max_connections(max_connections) - .min_connections(max_connections) + .min_connections(max_connections); + + if is_in_memory { + pool_options = pool_options.idle_timeout(None).max_lifetime(None); + } + + let pool = pool_options .connect_with(options) .await .map_err(|e| map_db_error(&e))?; @@ -37,30 +48,31 @@ impl SqliteStore { Ok(Self { pool }) } - pub async fn migrate(&self) -> Result<()> { - let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .join("migrations") - .join("sqlite"); - let migrator = sqlx::migrate::Migrator::new(path) - .await - .map_err(|e| map_migrate_error(&e))?; - migrator + pub async fn migrate(&self) -> PersistenceResult<()> { + SQLITE_MIGRATOR .run(&self.pool) .await .map_err(|e| map_migrate_error(&e)) } - pub async fn put(&self, object_type: &str, id: &str, name: &str, payload: &[u8]) -> Result<()> { + pub async fn put( + &self, + object_type: &str, + id: &str, + name: &str, + payload: &[u8], + labels: Option<&str>, + ) -> PersistenceResult<()> { let now_ms = current_time_ms()?; sqlx::query( r#" -INSERT INTO "objects" ("object_type", "id", "name", "payload", "created_at_ms", "updated_at_ms") -VALUES (?1, ?2, ?3, ?4, ?5, ?5) -ON CONFLICT ("id") DO UPDATE SET +INSERT INTO "objects" ("object_type", "id", "name", "payload", "created_at_ms", "updated_at_ms", "labels") +VALUES (?1, ?2, ?3, ?4, ?5, ?5, ?6) +ON CONFLICT ("object_type", "name") WHERE "name" IS NOT NULL DO UPDATE SET "payload" = excluded."payload", - "updated_at_ms" = excluded."updated_at_ms" -WHERE "objects"."object_type" = excluded."object_type" + "updated_at_ms" = excluded."updated_at_ms", + "labels" = excluded."labels" "#, ) .bind(object_type) @@ -68,16 +80,21 @@ WHERE "objects"."object_type" = excluded."object_type" .bind(name) .bind(payload) .bind(now_ms) + .bind(labels.unwrap_or("{}")) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; Ok(()) } - pub async fn get(&self, object_type: &str, id: &str) -> Result> { + pub async fn get( + &self, + object_type: &str, + id: &str, + ) -> PersistenceResult> { let row = sqlx::query( r#" -SELECT "object_type", "id", "name", "payload", "created_at_ms", "updated_at_ms" +SELECT "object_type", "id", "name", "payload", "created_at_ms", "updated_at_ms", "labels" FROM "objects" WHERE "object_type" = ?1 AND "id" = ?2 "#, @@ -88,20 +105,17 @@ WHERE "object_type" = ?1 AND "id" = ?2 .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(|row| ObjectRecord { - object_type: row.get("object_type"), - id: row.get("id"), - name: row.get("name"), - payload: row.get("payload"), - created_at_ms: row.get("created_at_ms"), - updated_at_ms: row.get("updated_at_ms"), - })) + Ok(row.map(row_to_object_record)) } - pub async fn get_by_name(&self, object_type: &str, name: &str) -> Result> { + pub async fn get_by_name( + &self, + object_type: &str, + name: &str, + ) -> PersistenceResult> { let row = sqlx::query( r#" -SELECT "object_type", "id", "name", "payload", "created_at_ms", "updated_at_ms" +SELECT "object_type", "id", "name", "payload", "created_at_ms", "updated_at_ms", "labels" FROM "objects" WHERE "object_type" = ?1 AND "name" = ?2 "#, @@ -112,17 +126,10 @@ WHERE "object_type" = ?1 AND "name" = ?2 .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(|row| ObjectRecord { - object_type: row.get("object_type"), - id: row.get("id"), - name: row.get("name"), - payload: row.get("payload"), - created_at_ms: row.get("created_at_ms"), - updated_at_ms: row.get("updated_at_ms"), - })) + Ok(row.map(row_to_object_record)) } - pub async fn delete(&self, object_type: &str, id: &str) -> Result { + pub async fn delete(&self, object_type: &str, id: &str) -> PersistenceResult { let result = sqlx::query( r#" DELETE FROM "objects" @@ -137,7 +144,7 @@ WHERE "object_type" = ?1 AND "id" = ?2 Ok(result.rows_affected() > 0) } - pub async fn delete_by_name(&self, object_type: &str, name: &str) -> Result { + pub async fn delete_by_name(&self, object_type: &str, name: &str) -> PersistenceResult { let result = sqlx::query( r#" DELETE FROM "objects" @@ -157,10 +164,10 @@ WHERE "object_type" = ?1 AND "name" = ?2 object_type: &str, limit: u32, offset: u32, - ) -> Result> { + ) -> PersistenceResult> { let rows = sqlx::query( r#" -SELECT "object_type", "id", "name", "payload", "created_at_ms", "updated_at_ms" +SELECT "object_type", "id", "name", "payload", "created_at_ms", "updated_at_ms", "labels" FROM "objects" WHERE "object_type" = ?1 ORDER BY "created_at_ms" ASC, "name" ASC @@ -174,25 +181,37 @@ LIMIT ?2 OFFSET ?3 .await .map_err(|e| map_db_error(&e))?; - let records = rows + Ok(rows.into_iter().map(row_to_object_record).collect()) + } + pub async fn list_with_selector( + &self, + object_type: &str, + label_selector: &str, + limit: u32, + offset: u32, + ) -> PersistenceResult> { + use super::parse_label_selector; + + let required_labels = parse_label_selector(label_selector)?; + let all_records = self.list(object_type, u32::MAX, 0).await?; + + let filtered: Vec = all_records .into_iter() - .map(|row| ObjectRecord { - object_type: row.get("object_type"), - id: row.get("id"), - name: row.get("name"), - payload: row.get("payload"), - created_at_ms: row.get("created_at_ms"), - updated_at_ms: row.get("updated_at_ms"), + .filter(|record| { + let labels_json = record.labels.as_deref().unwrap_or("{}"); + let labels: std::collections::HashMap = + serde_json::from_str(labels_json).unwrap_or_default(); + + required_labels + .iter() + .all(|(key, value)| labels.get(key).is_some_and(|v| v == value)) }) + .skip(offset as usize) + .take(limit as usize) .collect(); - Ok(records) + Ok(filtered) } - - // ------------------------------------------------------------------- - // Policy revision operations - // ------------------------------------------------------------------- - pub async fn put_policy_revision( &self, id: &str, @@ -200,19 +219,35 @@ LIMIT ?2 OFFSET ?3 version: i64, payload: &[u8], hash: &str, - ) -> Result<()> { + ) -> PersistenceResult<()> { let now_ms = current_time_ms()?; + let record = PolicyRecord { + id: id.to_string(), + sandbox_id: sandbox_id.to_string(), + version, + policy_payload: payload.to_vec(), + policy_hash: hash.to_string(), + status: "pending".to_string(), + load_error: None, + created_at_ms: now_ms, + loaded_at_ms: None, + }; + let wrapped_payload = policy_payload_from_record(&record)?; + sqlx::query( r#" -INSERT INTO "sandbox_policies" ("id", "sandbox_id", "version", "policy_payload", "policy_hash", "status", "created_at_ms") -VALUES (?1, ?2, ?3, ?4, ?5, 'pending', ?6) +INSERT INTO "objects" ( + "object_type", "id", "scope", "version", "status", "payload", "created_at_ms", "updated_at_ms" +) +VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?7) "#, ) + .bind(POLICY_OBJECT_TYPE) .bind(id) .bind(sandbox_id) .bind(version) - .bind(payload) - .bind(hash) + .bind("pending") + .bind(wrapped_payload) .bind(now_ms) .execute(&self.pool) .await @@ -220,61 +255,70 @@ VALUES (?1, ?2, ?3, ?4, ?5, 'pending', ?6) Ok(()) } - pub async fn get_latest_policy(&self, sandbox_id: &str) -> Result> { + pub async fn get_latest_policy( + &self, + sandbox_id: &str, + ) -> PersistenceResult> { let row = sqlx::query( r#" -SELECT "id", "sandbox_id", "version", "policy_payload", "policy_hash", "status", "load_error", "created_at_ms", "loaded_at_ms" -FROM "sandbox_policies" -WHERE "sandbox_id" = ?1 -ORDER BY "version" DESC +SELECT "id", "scope", "version", "status", "payload", "created_at_ms" +FROM "objects" +WHERE "object_type" = ?1 AND "scope" = ?2 +ORDER BY "version" DESC, "created_at_ms" DESC LIMIT 1 "#, ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .fetch_optional(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(row_to_policy_record)) + row.map(row_to_policy_record).transpose() } - pub async fn get_latest_loaded_policy(&self, sandbox_id: &str) -> Result> { + pub async fn get_latest_loaded_policy( + &self, + sandbox_id: &str, + ) -> PersistenceResult> { let row = sqlx::query( r#" -SELECT "id", "sandbox_id", "version", "policy_payload", "policy_hash", "status", "load_error", "created_at_ms", "loaded_at_ms" -FROM "sandbox_policies" -WHERE "sandbox_id" = ?1 AND "status" = 'loaded' -ORDER BY "version" DESC +SELECT "id", "scope", "version", "status", "payload", "created_at_ms" +FROM "objects" +WHERE "object_type" = ?1 AND "scope" = ?2 AND "status" = 'loaded' +ORDER BY "version" DESC, "created_at_ms" DESC LIMIT 1 "#, ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .fetch_optional(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(row_to_policy_record)) + row.map(row_to_policy_record).transpose() } pub async fn get_policy_by_version( &self, sandbox_id: &str, version: i64, - ) -> Result> { + ) -> PersistenceResult> { let row = sqlx::query( r#" -SELECT "id", "sandbox_id", "version", "policy_payload", "policy_hash", "status", "load_error", "created_at_ms", "loaded_at_ms" -FROM "sandbox_policies" -WHERE "sandbox_id" = ?1 AND "version" = ?2 +SELECT "id", "scope", "version", "status", "payload", "created_at_ms" +FROM "objects" +WHERE "object_type" = ?1 AND "scope" = ?2 AND "version" = ?3 "#, ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .bind(version) .fetch_optional(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(row_to_policy_record)) + row.map(row_to_policy_record).transpose() } pub async fn list_policies( @@ -282,16 +326,17 @@ WHERE "sandbox_id" = ?1 AND "version" = ?2 sandbox_id: &str, limit: u32, offset: u32, - ) -> Result> { + ) -> PersistenceResult> { let rows = sqlx::query( r#" -SELECT "id", "sandbox_id", "version", "policy_payload", "policy_hash", "status", "load_error", "created_at_ms", "loaded_at_ms" -FROM "sandbox_policies" -WHERE "sandbox_id" = ?1 -ORDER BY "version" DESC -LIMIT ?2 OFFSET ?3 +SELECT "id", "scope", "version", "status", "payload", "created_at_ms" +FROM "objects" +WHERE "object_type" = ?1 AND "scope" = ?2 +ORDER BY "version" DESC, "created_at_ms" DESC +LIMIT ?3 OFFSET ?4 "#, ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .bind(i64::from(limit)) .bind(i64::from(offset)) @@ -299,7 +344,7 @@ LIMIT ?2 OFFSET ?3 .await .map_err(|e| map_db_error(&e))?; - Ok(rows.into_iter().map(row_to_policy_record).collect()) + rows.into_iter().map(row_to_policy_record).collect() } pub async fn update_policy_status( @@ -309,19 +354,30 @@ LIMIT ?2 OFFSET ?3 status: &str, load_error: Option<&str>, loaded_at_ms: Option, - ) -> Result { + ) -> PersistenceResult { + let Some(mut record) = self.get_policy_by_version(sandbox_id, version).await? else { + return Ok(false); + }; + + record.status = status.to_string(); + record.load_error = load_error.map(ToOwned::to_owned); + record.loaded_at_ms = loaded_at_ms; + let payload = policy_payload_from_record(&record)?; + let now_ms = current_time_ms()?; + let result = sqlx::query( r#" -UPDATE "sandbox_policies" -SET "status" = ?3, "load_error" = ?4, "loaded_at_ms" = ?5 -WHERE "sandbox_id" = ?1 AND "version" = ?2 +UPDATE "objects" +SET "status" = ?4, "payload" = ?5, "updated_at_ms" = ?6 +WHERE "object_type" = ?1 AND "scope" = ?2 AND "version" = ?3 "#, ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .bind(version) .bind(status) - .bind(load_error) - .bind(loaded_at_ms) + .bind(payload) + .bind(now_ms) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; @@ -332,57 +388,48 @@ WHERE "sandbox_id" = ?1 AND "version" = ?2 &self, sandbox_id: &str, before_version: i64, - ) -> Result { + ) -> PersistenceResult { + let now_ms = current_time_ms()?; let result = sqlx::query( r#" -UPDATE "sandbox_policies" -SET "status" = 'superseded' -WHERE "sandbox_id" = ?1 AND "version" < ?2 AND "status" IN ('pending', 'loaded') +UPDATE "objects" +SET "status" = 'superseded', "updated_at_ms" = ?4 +WHERE "object_type" = ?1 + AND "scope" = ?2 + AND "version" < ?3 + AND "status" IN ('pending', 'loaded') "#, ) + .bind(POLICY_OBJECT_TYPE) .bind(sandbox_id) .bind(before_version) + .bind(now_ms) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; Ok(result.rows_affected()) } - // ------------------------------------------------------------------- - // Draft policy chunk operations - // ------------------------------------------------------------------- - - pub async fn put_draft_chunk(&self, chunk: &DraftChunkRecord) -> Result<()> { + pub async fn put_draft_chunk(&self, chunk: &DraftChunkRecord) -> PersistenceResult<()> { + let payload = draft_chunk_payload_from_record(chunk)?; sqlx::query( r#" -INSERT INTO "draft_policy_chunks" - ("id", "sandbox_id", "draft_version", "status", "rule_name", - "proposed_rule", "rationale", "security_notes", "confidence", - "created_at_ms", "decided_at_ms", "host", "port", "binary", - "hit_count", "first_seen_ms", "last_seen_ms") -VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17) -ON CONFLICT ("sandbox_id", "host", "port", "binary") - WHERE "status" IN ('pending', 'approved', 'rejected') -DO UPDATE SET - "hit_count" = "draft_policy_chunks"."hit_count" + excluded."hit_count", - "last_seen_ms" = excluded."last_seen_ms" +INSERT INTO "objects" ( + "object_type", "id", "scope", "status", "dedup_key", "hit_count", "payload", "created_at_ms", "updated_at_ms" +) +VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9) +ON CONFLICT ("object_type", "scope", "dedup_key") WHERE "dedup_key" IS NOT NULL DO UPDATE SET + "hit_count" = "objects"."hit_count" + excluded."hit_count", + "updated_at_ms" = excluded."updated_at_ms" "#, ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(&chunk.id) .bind(&chunk.sandbox_id) - .bind(chunk.draft_version) .bind(&chunk.status) - .bind(&chunk.rule_name) - .bind(&chunk.proposed_rule) - .bind(&chunk.rationale) - .bind(&chunk.security_notes) - .bind(chunk.confidence) - .bind(chunk.created_at_ms) - .bind(chunk.decided_at_ms) - .bind(&chunk.host) - .bind(chunk.port) - .bind(&chunk.binary) - .bind(chunk.hit_count) + .bind(draft_chunk_dedup_key(chunk)) + .bind(i64::from(chunk.hit_count)) + .bind(payload) .bind(chunk.first_seen_ms) .bind(chunk.last_seen_ms) .execute(&self.pool) @@ -391,33 +438,38 @@ DO UPDATE SET Ok(()) } - pub async fn get_draft_chunk(&self, id: &str) -> Result> { + pub async fn get_draft_chunk(&self, id: &str) -> PersistenceResult> { let row = sqlx::query( r#" -SELECT * FROM "draft_policy_chunks" WHERE "id" = ?1 +SELECT "id", "scope", "status", "hit_count", "payload", "created_at_ms", "updated_at_ms" +FROM "objects" +WHERE "object_type" = ?1 AND "id" = ?2 "#, ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(id) .fetch_optional(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.map(row_to_draft_chunk_record)) + row.map(row_to_draft_chunk_record).transpose() } pub async fn list_draft_chunks( &self, sandbox_id: &str, status_filter: Option<&str>, - ) -> Result> { + ) -> PersistenceResult> { let rows = if let Some(status) = status_filter { sqlx::query( r#" -SELECT * FROM "draft_policy_chunks" -WHERE "sandbox_id" = ?1 AND "status" = ?2 +SELECT "id", "scope", "status", "hit_count", "payload", "created_at_ms", "updated_at_ms" +FROM "objects" +WHERE "object_type" = ?1 AND "scope" = ?2 AND "status" = ?3 ORDER BY "created_at_ms" DESC "#, ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(sandbox_id) .bind(status) .fetch_all(&self.pool) @@ -425,18 +477,20 @@ ORDER BY "created_at_ms" DESC } else { sqlx::query( r#" -SELECT * FROM "draft_policy_chunks" -WHERE "sandbox_id" = ?1 +SELECT "id", "scope", "status", "hit_count", "payload", "created_at_ms", "updated_at_ms" +FROM "objects" +WHERE "object_type" = ?1 AND "scope" = ?2 ORDER BY "created_at_ms" DESC "#, ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(sandbox_id) .fetch_all(&self.pool) .await } .map_err(|e| map_db_error(&e))?; - Ok(rows.into_iter().map(row_to_draft_chunk_record).collect()) + rows.into_iter().map(row_to_draft_chunk_record).collect() } pub async fn update_draft_chunk_status( @@ -444,46 +498,80 @@ ORDER BY "created_at_ms" DESC id: &str, status: &str, decided_at_ms: Option, - ) -> Result { + ) -> PersistenceResult { + let Some(mut record) = self.get_draft_chunk(id).await? else { + return Ok(false); + }; + + record.status = status.to_string(); + record.decided_at_ms = decided_at_ms; + record.last_seen_ms = current_time_ms()?; + let payload = draft_chunk_payload_from_record(&record)?; + let result = sqlx::query( r#" -UPDATE "draft_policy_chunks" -SET "status" = ?2, "decided_at_ms" = ?3 -WHERE "id" = ?1 +UPDATE "objects" +SET "status" = ?3, "payload" = ?4, "updated_at_ms" = ?5 +WHERE "object_type" = ?1 AND "id" = ?2 "#, ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(id) .bind(status) - .bind(decided_at_ms) + .bind(payload) + .bind(record.last_seen_ms) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; Ok(result.rows_affected() > 0) } - pub async fn update_draft_chunk_rule(&self, id: &str, proposed_rule: &[u8]) -> Result { + pub async fn update_draft_chunk_rule( + &self, + id: &str, + proposed_rule: &[u8], + ) -> PersistenceResult { + let Some(mut record) = self.get_draft_chunk(id).await? else { + return Ok(false); + }; + + if record.status != "pending" { + return Ok(false); + } + + record.proposed_rule = proposed_rule.to_vec(); + record.last_seen_ms = current_time_ms()?; + let payload = draft_chunk_payload_from_record(&record)?; + let result = sqlx::query( r#" -UPDATE "draft_policy_chunks" -SET "proposed_rule" = ?2 -WHERE "id" = ?1 AND "status" = 'pending' +UPDATE "objects" +SET "payload" = ?3, "updated_at_ms" = ?4 +WHERE "object_type" = ?1 AND "id" = ?2 AND "status" = 'pending' "#, ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(id) - .bind(proposed_rule) + .bind(payload) + .bind(record.last_seen_ms) .execute(&self.pool) .await .map_err(|e| map_db_error(&e))?; Ok(result.rows_affected() > 0) } - pub async fn delete_draft_chunks(&self, sandbox_id: &str, status: &str) -> Result { + pub async fn delete_draft_chunks( + &self, + sandbox_id: &str, + status: &str, + ) -> PersistenceResult { let result = sqlx::query( r#" -DELETE FROM "draft_policy_chunks" -WHERE "sandbox_id" = ?1 AND "status" = ?2 +DELETE FROM "objects" +WHERE "object_type" = ?1 AND "scope" = ?2 AND "status" = ?3 "#, ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(sandbox_id) .bind(status) .execute(&self.pool) @@ -492,55 +580,79 @@ WHERE "sandbox_id" = ?1 AND "status" = ?2 Ok(result.rows_affected()) } - pub async fn get_draft_version(&self, sandbox_id: &str) -> Result { - let row = sqlx::query( + pub async fn get_draft_version(&self, sandbox_id: &str) -> PersistenceResult { + let rows = sqlx::query( r#" -SELECT COALESCE(MAX("draft_version"), 0) as "max_version" -FROM "draft_policy_chunks" -WHERE "sandbox_id" = ?1 +SELECT "payload" +FROM "objects" +WHERE "object_type" = ?1 AND "scope" = ?2 "#, ) + .bind(DRAFT_CHUNK_OBJECT_TYPE) .bind(sandbox_id) - .fetch_one(&self.pool) + .fetch_all(&self.pool) .await .map_err(|e| map_db_error(&e))?; - Ok(row.get("max_version")) + let mut max_version = 0_i64; + for row in rows { + let payload: Vec = row.get("payload"); + let wrapper = draft_chunk_record_from_parts( + String::new(), + sandbox_id.to_string(), + String::new(), + 0, + &payload, + 0, + 0, + )?; + max_version = max_version.max(wrapper.draft_version); + } + Ok(max_version) } } -fn row_to_draft_chunk_record(row: sqlx::sqlite::SqliteRow) -> DraftChunkRecord { - DraftChunkRecord { - id: row.get("id"), - sandbox_id: row.get("sandbox_id"), - draft_version: row.get("draft_version"), - status: row.get("status"), - rule_name: row.get("rule_name"), - proposed_rule: row.get("proposed_rule"), - rationale: row.get("rationale"), - security_notes: row.get("security_notes"), - confidence: row.get("confidence"), - created_at_ms: row.get("created_at_ms"), - decided_at_ms: row.get("decided_at_ms"), - host: row.get("host"), - port: row.get("port"), - binary: row.get("binary"), - hit_count: row.get("hit_count"), - first_seen_ms: row.get("first_seen_ms"), - last_seen_ms: row.get("last_seen_ms"), - } +fn draft_chunk_dedup_key(chunk: &DraftChunkRecord) -> String { + format!("{}|{}|{}", chunk.host, chunk.port, chunk.binary) } -fn row_to_policy_record(row: sqlx::sqlite::SqliteRow) -> PolicyRecord { - PolicyRecord { +fn row_to_object_record(row: sqlx::sqlite::SqliteRow) -> ObjectRecord { + ObjectRecord { + object_type: row.get("object_type"), id: row.get("id"), - sandbox_id: row.get("sandbox_id"), - version: row.get("version"), - policy_payload: row.get("policy_payload"), - policy_hash: row.get("policy_hash"), - status: row.get("status"), - load_error: row.get("load_error"), + name: row.get("name"), + payload: row.get("payload"), created_at_ms: row.get("created_at_ms"), - loaded_at_ms: row.get("loaded_at_ms"), + updated_at_ms: row.get("updated_at_ms"), + labels: row.get("labels"), } } + +fn row_to_policy_record(row: sqlx::sqlite::SqliteRow) -> PersistenceResult { + let id: String = row.get("id"); + let sandbox_id: String = row.get("scope"); + let version: i64 = row.get("version"); + let status: String = row.get("status"); + let payload: Vec = row.get("payload"); + let created_at_ms: i64 = row.get("created_at_ms"); + policy_record_from_parts(id, sandbox_id, version, status, &payload, created_at_ms) +} + +fn row_to_draft_chunk_record(row: sqlx::sqlite::SqliteRow) -> PersistenceResult { + let id: String = row.get("id"); + let sandbox_id: String = row.get("scope"); + let status: String = row.get("status"); + let hit_count: i64 = row.get("hit_count"); + let payload: Vec = row.get("payload"); + let created_at_ms: i64 = row.get("created_at_ms"); + let updated_at_ms: i64 = row.get("updated_at_ms"); + draft_chunk_record_from_parts( + id, + sandbox_id, + status, + hit_count, + &payload, + created_at_ms, + updated_at_ms, + ) +} diff --git a/crates/openshell-server/src/persistence/tests.rs b/crates/openshell-server/src/persistence/tests.rs index a5223a9f45..bef95d4b65 100644 --- a/crates/openshell-server/src/persistence/tests.rs +++ b/crates/openshell-server/src/persistence/tests.rs @@ -1,8 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -use super::{ObjectId, ObjectName, ObjectType, Store, generate_name}; -use openshell_core::proto::ObjectForTest; +use super::{ObjectType, Store, generate_name}; +use crate::policy_store::PolicyStoreExt; +use openshell_core::proto::{ObjectForTest, SandboxPolicy}; +use prost::Message; #[tokio::test] async fn sqlite_put_get_round_trip() { @@ -11,7 +13,7 @@ async fn sqlite_put_get_round_trip() { .unwrap(); store - .put("sandbox", "abc", "my-sandbox", b"payload") + .put("sandbox", "abc", "my-sandbox", b"payload", None) .await .unwrap(); @@ -22,6 +24,16 @@ async fn sqlite_put_get_round_trip() { assert_eq!(record.payload, b"payload"); } +#[tokio::test] +async fn sqlite_connect_runs_embedded_migrations() { + let store = Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(); + + let records = store.list("sandbox", 10, 0).await.unwrap(); + assert!(records.is_empty()); +} + #[tokio::test] async fn sqlite_updates_timestamp() { let store = Store::connect("sqlite::memory:?cache=shared") @@ -29,14 +41,14 @@ async fn sqlite_updates_timestamp() { .unwrap(); store - .put("sandbox", "abc", "my-sandbox", b"payload") + .put("sandbox", "abc", "my-sandbox", b"payload", None) .await .unwrap(); let first = store.get("sandbox", "abc").await.unwrap().unwrap(); store - .put("sandbox", "abc", "my-sandbox", b"payload2") + .put("sandbox", "abc", "my-sandbox", b"payload2", None) .await .unwrap(); @@ -56,7 +68,7 @@ async fn sqlite_list_paging() { let name = format!("name-{idx}"); let payload = format!("payload-{idx}"); store - .put("sandbox", &id, &name, payload.as_bytes()) + .put("sandbox", &id, &name, payload.as_bytes(), None) .await .unwrap(); } @@ -74,7 +86,7 @@ async fn sqlite_delete_behavior() { .unwrap(); store - .put("sandbox", "abc", "my-sandbox", b"payload") + .put("sandbox", "abc", "my-sandbox", b"payload", None) .await .unwrap(); @@ -117,7 +129,7 @@ async fn sqlite_get_by_name() { .unwrap(); store - .put("sandbox", "id-1", "my-sandbox", b"payload") + .put("sandbox", "id-1", "my-sandbox", b"payload", None) .await .unwrap(); @@ -171,7 +183,7 @@ async fn sqlite_delete_by_name() { .unwrap(); store - .put("sandbox", "id-1", "my-sandbox", b"payload") + .put("sandbox", "id-1", "my-sandbox", b"payload", None) .await .unwrap(); @@ -192,19 +204,27 @@ async fn sqlite_name_unique_per_object_type() { .unwrap(); store - .put("sandbox", "id-1", "shared-name", b"payload1") + .put("sandbox", "id-1", "shared-name", b"payload1", None) .await .unwrap(); - // Same name, same object_type, different id -> should fail (unique constraint). - let result = store - .put("sandbox", "id-2", "shared-name", b"payload2") - .await; - assert!(result.is_err()); + // Same name, same object_type, different id -> upsert on name. + store + .put("sandbox", "id-2", "shared-name", b"payload2", None) + .await + .unwrap(); + + let record = store + .get_by_name("sandbox", "shared-name") + .await + .unwrap() + .unwrap(); + assert_eq!(record.id, "id-1"); + assert_eq!(record.payload, b"payload2"); // Same name, different object_type -> should succeed. store - .put("secret", "id-3", "shared-name", b"payload3") + .put("secret", "id-3", "shared-name", b"payload3", None) .await .unwrap(); } @@ -216,17 +236,16 @@ async fn sqlite_id_globally_unique() { .unwrap(); store - .put("sandbox", "same-id", "name-a", b"payload1") + .put("sandbox", "same-id", "name-a", b"payload1", None) .await .unwrap(); - // Same id, different object_type -> the upsert is a no-op (WHERE - // clause prevents updating a row with a different object_type). - // The original row is preserved unchanged. - store - .put("secret", "same-id", "name-b", b"payload2") - .await - .unwrap(); + // Same id, different object_type -> should fail because ids remain global + // primary keys even when writes upsert on name. + let result = store + .put("secret", "same-id", "name-b", b"payload2", None) + .await; + assert!(result.is_err()); // Original row is untouched. let record = store.get("sandbox", "same-id").await.unwrap().unwrap(); @@ -253,16 +272,192 @@ impl ObjectType for ObjectForTest { } } -impl ObjectId for ObjectForTest { - fn object_id(&self) -> &str { - &self.id - } +// ObjectId, ObjectName, ObjectLabels implementations +// for ObjectForTest are in openshell-core::metadata + +// --------------------------------------------------------------------------- +// ObjectMeta tests (labels) +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn labels_round_trip() { + let store = Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(); + + let labels = r#"{"env":"production","team":"platform"}"#; + store + .put( + "sandbox", + "id-1", + "labeled-sandbox", + b"payload", + Some(labels), + ) + .await + .unwrap(); + + let record = store.get("sandbox", "id-1").await.unwrap().unwrap(); + assert_eq!(record.labels.as_deref(), Some(labels)); +} + +#[tokio::test] +async fn label_selector_single_match() { + let store = Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(); + + store + .put("sandbox", "id-1", "s1", b"p1", Some(r#"{"env":"prod"}"#)) + .await + .unwrap(); + store + .put("sandbox", "id-2", "s2", b"p2", Some(r#"{"env":"dev"}"#)) + .await + .unwrap(); + store + .put( + "sandbox", + "id-3", + "s3", + b"p3", + Some(r#"{"env":"prod","team":"platform"}"#), + ) + .await + .unwrap(); + + let results = store + .list_with_selector("sandbox", "env=prod", 10, 0) + .await + .unwrap(); + + assert_eq!(results.len(), 2); + let ids: Vec<&str> = results.iter().map(|r| r.id.as_str()).collect(); + assert!(ids.contains(&"id-1")); + assert!(ids.contains(&"id-3")); +} + +#[tokio::test] +async fn label_selector_multiple_labels() { + let store = Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(); + + store + .put( + "sandbox", + "id-1", + "s1", + b"p1", + Some(r#"{"env":"prod","team":"platform"}"#), + ) + .await + .unwrap(); + store + .put( + "sandbox", + "id-2", + "s2", + b"p2", + Some(r#"{"env":"prod","team":"data"}"#), + ) + .await + .unwrap(); + store + .put( + "sandbox", + "id-3", + "s3", + b"p3", + Some(r#"{"env":"dev","team":"platform"}"#), + ) + .await + .unwrap(); + + let results = store + .list_with_selector("sandbox", "env=prod,team=platform", 10, 0) + .await + .unwrap(); + + assert_eq!(results.len(), 1); + assert_eq!(results[0].id, "id-1"); +} + +#[tokio::test] +async fn label_selector_no_match() { + let store = Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(); + + store + .put("sandbox", "id-1", "s1", b"p1", Some(r#"{"env":"prod"}"#)) + .await + .unwrap(); + + let results = store + .list_with_selector("sandbox", "env=staging", 10, 0) + .await + .unwrap(); + + assert_eq!(results.len(), 0); } -impl ObjectName for ObjectForTest { - fn object_name(&self) -> &str { - &self.name +#[tokio::test] +async fn label_selector_respects_paging() { + let store = Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(); + + for idx in 0..5 { + let id = format!("id-{idx}"); + let name = format!("name-{idx}"); + store + .put("sandbox", &id, &name, b"payload", Some(r#"{"env":"prod"}"#)) + .await + .unwrap(); } + + let page1 = store + .list_with_selector("sandbox", "env=prod", 2, 0) + .await + .unwrap(); + assert_eq!(page1.len(), 2); + + let page2 = store + .list_with_selector("sandbox", "env=prod", 2, 2) + .await + .unwrap(); + assert_eq!(page2.len(), 2); + + let page3 = store + .list_with_selector("sandbox", "env=prod", 2, 4) + .await + .unwrap(); + assert_eq!(page3.len(), 1); +} + +#[tokio::test] +async fn empty_labels_not_matched_by_selector() { + let store = Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(); + + store + .put("sandbox", "id-1", "s1", b"p1", None) + .await + .unwrap(); + store + .put("sandbox", "id-2", "s2", b"p2", Some(r#"{"env":"prod"}"#)) + .await + .unwrap(); + + let results = store + .list_with_selector("sandbox", "env=prod", 10, 0) + .await + .unwrap(); + + assert_eq!(results.len(), 1); + assert_eq!(results[0].id, "id-2"); } // --------------------------------------------------------------------------- @@ -275,8 +470,9 @@ async fn policy_put_and_get_latest() { .await .unwrap(); + let policy_v1 = SandboxPolicy::default().encode_to_vec(); store - .put_policy_revision("p1", "sandbox-1", 1, b"policy-v1", "hash1") + .put_policy_revision("p1", "sandbox-1", 1, &policy_v1, "hash1") .await .unwrap(); @@ -284,11 +480,16 @@ async fn policy_put_and_get_latest() { assert_eq!(latest.version, 1); assert_eq!(latest.policy_hash, "hash1"); assert_eq!(latest.status, "pending"); - assert_eq!(latest.policy_payload, b"policy-v1"); + assert_eq!(latest.policy_payload, policy_v1); // Add version 2 + let policy_v2 = SandboxPolicy { + version: 2, + ..SandboxPolicy::default() + } + .encode_to_vec(); store - .put_policy_revision("p2", "sandbox-1", 2, b"policy-v2", "hash2") + .put_policy_revision("p2", "sandbox-1", 2, &policy_v2, "hash2") .await .unwrap(); @@ -303,12 +504,18 @@ async fn policy_get_by_version() { .await .unwrap(); + let policy_v1 = SandboxPolicy::default().encode_to_vec(); + let policy_v2 = SandboxPolicy { + version: 2, + ..SandboxPolicy::default() + } + .encode_to_vec(); store - .put_policy_revision("p1", "sandbox-1", 1, b"v1", "h1") + .put_policy_revision("p1", "sandbox-1", 1, &policy_v1, "h1") .await .unwrap(); store - .put_policy_revision("p2", "sandbox-1", 2, b"v2", "h2") + .put_policy_revision("p2", "sandbox-1", 2, &policy_v2, "h2") .await .unwrap(); @@ -338,8 +545,9 @@ async fn policy_update_status_and_get_loaded() { .await .unwrap(); + let payload = SandboxPolicy::default().encode_to_vec(); store - .put_policy_revision("p1", "sandbox-1", 1, b"v1", "h1") + .put_policy_revision("p1", "sandbox-1", 1, &payload, "h1") .await .unwrap(); @@ -370,8 +578,9 @@ async fn policy_status_failed_with_error() { .await .unwrap(); + let payload = SandboxPolicy::default().encode_to_vec(); store - .put_policy_revision("p1", "sandbox-1", 1, b"v1", "h1") + .put_policy_revision("p1", "sandbox-1", 1, &payload, "h1") .await .unwrap(); @@ -395,16 +604,17 @@ async fn policy_supersede_older() { .await .unwrap(); + let payload = SandboxPolicy::default().encode_to_vec(); store - .put_policy_revision("p1", "sandbox-1", 1, b"v1", "h1") + .put_policy_revision("p1", "sandbox-1", 1, &payload, "h1") .await .unwrap(); store - .put_policy_revision("p2", "sandbox-1", 2, b"v2", "h2") + .put_policy_revision("p2", "sandbox-1", 2, &payload, "h2") .await .unwrap(); store - .put_policy_revision("p3", "sandbox-1", 3, b"v3", "h3") + .put_policy_revision("p3", "sandbox-1", 3, &payload, "h3") .await .unwrap(); @@ -449,16 +659,17 @@ async fn policy_list_ordered_by_version_desc() { .await .unwrap(); + let payload = SandboxPolicy::default().encode_to_vec(); store - .put_policy_revision("p1", "sandbox-1", 1, b"v1", "h1") + .put_policy_revision("p1", "sandbox-1", 1, &payload, "h1") .await .unwrap(); store - .put_policy_revision("p2", "sandbox-1", 2, b"v2", "h2") + .put_policy_revision("p2", "sandbox-1", 2, &payload, "h2") .await .unwrap(); store - .put_policy_revision("p3", "sandbox-1", 3, b"v3", "h3") + .put_policy_revision("p3", "sandbox-1", 3, &payload, "h3") .await .unwrap(); @@ -481,18 +692,96 @@ async fn policy_isolation_between_sandboxes() { .await .unwrap(); + let policy_s1 = SandboxPolicy::default().encode_to_vec(); + let policy_s2 = SandboxPolicy { + version: 7, + ..SandboxPolicy::default() + } + .encode_to_vec(); store - .put_policy_revision("p1", "sandbox-1", 1, b"v1", "h1") + .put_policy_revision("p1", "sandbox-1", 1, &policy_s1, "h1") .await .unwrap(); store - .put_policy_revision("p2", "sandbox-2", 1, b"v1-s2", "h2") + .put_policy_revision("p2", "sandbox-2", 1, &policy_s2, "h2") .await .unwrap(); let s1 = store.get_latest_policy("sandbox-1").await.unwrap().unwrap(); let s2 = store.get_latest_policy("sandbox-2").await.unwrap().unwrap(); - assert_eq!(s1.policy_payload, b"v1"); - assert_eq!(s2.policy_payload, b"v1-s2"); + assert_eq!(s1.policy_payload, policy_s1); + assert_eq!(s2.policy_payload, policy_s2); +} + +// ---- Label selector parsing tests ---- + +#[test] +fn parse_label_selector_empty_string() { + let result = super::parse_label_selector("").unwrap(); + assert!(result.is_empty()); +} + +#[test] +fn parse_label_selector_single_pair() { + let result = super::parse_label_selector("env=prod").unwrap(); + assert_eq!(result.len(), 1); + assert_eq!(result.get("env"), Some(&"prod".to_string())); +} + +#[test] +fn parse_label_selector_multiple_pairs() { + let result = super::parse_label_selector("env=prod,tier=frontend,version=v1").unwrap(); + assert_eq!(result.len(), 3); + assert_eq!(result.get("env"), Some(&"prod".to_string())); + assert_eq!(result.get("tier"), Some(&"frontend".to_string())); + assert_eq!(result.get("version"), Some(&"v1".to_string())); +} + +#[test] +fn parse_label_selector_accepts_empty_value() { + // Kubernetes allows empty label values, so selectors should accept "key=" format + let result = super::parse_label_selector("env=").unwrap(); + assert_eq!(result.len(), 1); + assert_eq!(result.get("env"), Some(&String::new())); +} + +#[test] +fn parse_label_selector_multiple_with_empty_value() { + let result = super::parse_label_selector("env=,tier=frontend").unwrap(); + assert_eq!(result.len(), 2); + assert_eq!(result.get("env"), Some(&String::new())); + assert_eq!(result.get("tier"), Some(&"frontend".to_string())); +} + +#[test] +fn parse_label_selector_rejects_empty_key() { + let result = super::parse_label_selector("=value"); + assert!(result.is_err()); + assert!( + result + .unwrap_err() + .to_string() + .contains("key cannot be empty") + ); +} + +#[test] +fn parse_label_selector_rejects_missing_equals() { + let result = super::parse_label_selector("env"); + assert!(result.is_err()); + assert!( + result + .unwrap_err() + .to_string() + .contains("expected 'key=value'") + ); +} + +#[test] +fn parse_label_selector_handles_whitespace() { + let result = super::parse_label_selector("env = prod , tier = frontend").unwrap(); + assert_eq!(result.len(), 2); + assert_eq!(result.get("env"), Some(&"prod".to_string())); + assert_eq!(result.get("tier"), Some(&"frontend".to_string())); } diff --git a/crates/openshell-server/src/policy_store.rs b/crates/openshell-server/src/policy_store.rs new file mode 100644 index 0000000000..f0a43698e8 --- /dev/null +++ b/crates/openshell-server/src/policy_store.rs @@ -0,0 +1,369 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use crate::persistence::{DraftChunkRecord, PersistenceResult, PolicyRecord, Store}; +use openshell_core::proto::{ + DraftChunkPayload, NetworkPolicyRule, PolicyRevisionPayload, + SandboxPolicy as ProtoSandboxPolicy, +}; +use prost::Message; + +pub trait PolicyStoreExt { + async fn put_policy_revision( + &self, + id: &str, + sandbox_id: &str, + version: i64, + payload: &[u8], + hash: &str, + ) -> PersistenceResult<()>; + + async fn get_latest_policy(&self, sandbox_id: &str) -> PersistenceResult>; + + #[allow(dead_code)] + async fn get_latest_loaded_policy( + &self, + sandbox_id: &str, + ) -> PersistenceResult>; + + async fn get_policy_by_version( + &self, + sandbox_id: &str, + version: i64, + ) -> PersistenceResult>; + + async fn list_policies( + &self, + sandbox_id: &str, + limit: u32, + offset: u32, + ) -> PersistenceResult>; + + async fn update_policy_status( + &self, + sandbox_id: &str, + version: i64, + status: &str, + load_error: Option<&str>, + loaded_at_ms: Option, + ) -> PersistenceResult; + + async fn supersede_older_policies( + &self, + sandbox_id: &str, + before_version: i64, + ) -> PersistenceResult; + + async fn put_draft_chunk(&self, chunk: &DraftChunkRecord) -> PersistenceResult<()>; + + async fn get_draft_chunk(&self, id: &str) -> PersistenceResult>; + + async fn list_draft_chunks( + &self, + sandbox_id: &str, + status_filter: Option<&str>, + ) -> PersistenceResult>; + + async fn update_draft_chunk_status( + &self, + id: &str, + status: &str, + decided_at_ms: Option, + ) -> PersistenceResult; + + async fn update_draft_chunk_rule( + &self, + id: &str, + proposed_rule: &[u8], + ) -> PersistenceResult; + + async fn delete_draft_chunks(&self, sandbox_id: &str, status: &str) -> PersistenceResult; + + async fn get_draft_version(&self, sandbox_id: &str) -> PersistenceResult; +} + +impl PolicyStoreExt for Store { + async fn put_policy_revision( + &self, + id: &str, + sandbox_id: &str, + version: i64, + payload: &[u8], + hash: &str, + ) -> PersistenceResult<()> { + match self { + Self::Postgres(store) => { + store + .put_policy_revision(id, sandbox_id, version, payload, hash) + .await + } + Self::Sqlite(store) => { + store + .put_policy_revision(id, sandbox_id, version, payload, hash) + .await + } + } + } + + async fn get_latest_policy(&self, sandbox_id: &str) -> PersistenceResult> { + match self { + Self::Postgres(store) => store.get_latest_policy(sandbox_id).await, + Self::Sqlite(store) => store.get_latest_policy(sandbox_id).await, + } + } + + async fn get_latest_loaded_policy( + &self, + sandbox_id: &str, + ) -> PersistenceResult> { + match self { + Self::Postgres(store) => store.get_latest_loaded_policy(sandbox_id).await, + Self::Sqlite(store) => store.get_latest_loaded_policy(sandbox_id).await, + } + } + + async fn get_policy_by_version( + &self, + sandbox_id: &str, + version: i64, + ) -> PersistenceResult> { + match self { + Self::Postgres(store) => store.get_policy_by_version(sandbox_id, version).await, + Self::Sqlite(store) => store.get_policy_by_version(sandbox_id, version).await, + } + } + + async fn list_policies( + &self, + sandbox_id: &str, + limit: u32, + offset: u32, + ) -> PersistenceResult> { + match self { + Self::Postgres(store) => store.list_policies(sandbox_id, limit, offset).await, + Self::Sqlite(store) => store.list_policies(sandbox_id, limit, offset).await, + } + } + + async fn update_policy_status( + &self, + sandbox_id: &str, + version: i64, + status: &str, + load_error: Option<&str>, + loaded_at_ms: Option, + ) -> PersistenceResult { + match self { + Self::Postgres(store) => { + store + .update_policy_status(sandbox_id, version, status, load_error, loaded_at_ms) + .await + } + Self::Sqlite(store) => { + store + .update_policy_status(sandbox_id, version, status, load_error, loaded_at_ms) + .await + } + } + } + + async fn supersede_older_policies( + &self, + sandbox_id: &str, + before_version: i64, + ) -> PersistenceResult { + match self { + Self::Postgres(store) => { + store + .supersede_older_policies(sandbox_id, before_version) + .await + } + Self::Sqlite(store) => { + store + .supersede_older_policies(sandbox_id, before_version) + .await + } + } + } + + async fn put_draft_chunk(&self, chunk: &DraftChunkRecord) -> PersistenceResult<()> { + match self { + Self::Postgres(store) => store.put_draft_chunk(chunk).await, + Self::Sqlite(store) => store.put_draft_chunk(chunk).await, + } + } + + async fn get_draft_chunk(&self, id: &str) -> PersistenceResult> { + match self { + Self::Postgres(store) => store.get_draft_chunk(id).await, + Self::Sqlite(store) => store.get_draft_chunk(id).await, + } + } + + async fn list_draft_chunks( + &self, + sandbox_id: &str, + status_filter: Option<&str>, + ) -> PersistenceResult> { + match self { + Self::Postgres(store) => store.list_draft_chunks(sandbox_id, status_filter).await, + Self::Sqlite(store) => store.list_draft_chunks(sandbox_id, status_filter).await, + } + } + + async fn update_draft_chunk_status( + &self, + id: &str, + status: &str, + decided_at_ms: Option, + ) -> PersistenceResult { + match self { + Self::Postgres(store) => { + store + .update_draft_chunk_status(id, status, decided_at_ms) + .await + } + Self::Sqlite(store) => { + store + .update_draft_chunk_status(id, status, decided_at_ms) + .await + } + } + } + + async fn update_draft_chunk_rule( + &self, + id: &str, + proposed_rule: &[u8], + ) -> PersistenceResult { + match self { + Self::Postgres(store) => store.update_draft_chunk_rule(id, proposed_rule).await, + Self::Sqlite(store) => store.update_draft_chunk_rule(id, proposed_rule).await, + } + } + + async fn delete_draft_chunks(&self, sandbox_id: &str, status: &str) -> PersistenceResult { + match self { + Self::Postgres(store) => store.delete_draft_chunks(sandbox_id, status).await, + Self::Sqlite(store) => store.delete_draft_chunks(sandbox_id, status).await, + } + } + + async fn get_draft_version(&self, sandbox_id: &str) -> PersistenceResult { + match self { + Self::Postgres(store) => store.get_draft_version(sandbox_id).await, + Self::Sqlite(store) => store.get_draft_version(sandbox_id).await, + } + } +} + +pub fn policy_payload_from_record(record: &PolicyRecord) -> PersistenceResult> { + let policy = ProtoSandboxPolicy::decode(record.policy_payload.as_slice()).map_err(|e| { + crate::persistence::PersistenceError::Decode(format!("decode policy payload failed: {e}")) + })?; + Ok(PolicyRevisionPayload { + policy: Some(policy), + hash: record.policy_hash.clone(), + load_error: record.load_error.clone().unwrap_or_default(), + loaded_at_ms: record.loaded_at_ms.unwrap_or(0), + } + .encode_to_vec()) +} + +pub fn policy_record_from_parts( + id: String, + sandbox_id: String, + version: i64, + status: String, + payload: &[u8], + created_at_ms: i64, +) -> PersistenceResult { + let wrapper = PolicyRevisionPayload::decode(payload).map_err(|e| { + crate::persistence::PersistenceError::Decode(format!("decode policy wrapper failed: {e}")) + })?; + let policy = wrapper.policy.ok_or_else(|| { + crate::persistence::PersistenceError::Decode("policy wrapper missing policy".to_string()) + })?; + Ok(PolicyRecord { + id, + sandbox_id, + version, + policy_payload: policy.encode_to_vec(), + policy_hash: wrapper.hash, + status, + load_error: if wrapper.load_error.is_empty() { + None + } else { + Some(wrapper.load_error) + }, + created_at_ms, + loaded_at_ms: (wrapper.loaded_at_ms > 0).then_some(wrapper.loaded_at_ms), + }) +} + +pub fn draft_chunk_payload_from_record(chunk: &DraftChunkRecord) -> PersistenceResult> { + let proposed_rule = if chunk.proposed_rule.is_empty() { + None + } else { + Some( + NetworkPolicyRule::decode(chunk.proposed_rule.as_slice()).map_err(|e| { + crate::persistence::PersistenceError::Decode(format!( + "decode draft rule failed: {e}" + )) + })?, + ) + }; + Ok(DraftChunkPayload { + rule_name: chunk.rule_name.clone(), + proposed_rule, + rationale: chunk.rationale.clone(), + security_notes: chunk.security_notes.clone(), + #[allow(clippy::cast_possible_truncation)] // f64->f32 for confidence scores + confidence: chunk.confidence as f32, + decided_at_ms: chunk.decided_at_ms.unwrap_or(0), + host: chunk.host.clone(), + port: chunk.port, + binary: chunk.binary.clone(), + draft_version: chunk.draft_version, + } + .encode_to_vec()) +} + +pub fn draft_chunk_record_from_parts( + id: String, + sandbox_id: String, + status: String, + hit_count: i64, + payload: &[u8], + created_at_ms: i64, + updated_at_ms: i64, +) -> PersistenceResult { + let wrapper = DraftChunkPayload::decode(payload).map_err(|e| { + crate::persistence::PersistenceError::Decode(format!( + "decode draft chunk wrapper failed: {e}" + )) + })?; + let proposed_rule = wrapper + .proposed_rule + .map(|rule| rule.encode_to_vec()) + .unwrap_or_default(); + Ok(DraftChunkRecord { + id, + sandbox_id, + draft_version: wrapper.draft_version, + status, + rule_name: wrapper.rule_name, + proposed_rule, + rationale: wrapper.rationale, + security_notes: wrapper.security_notes, + confidence: f64::from(wrapper.confidence), + created_at_ms, + decided_at_ms: (wrapper.decided_at_ms > 0).then_some(wrapper.decided_at_ms), + host: wrapper.host, + port: wrapper.port, + binary: wrapper.binary, + hit_count: i32::try_from(hit_count).unwrap_or(i32::MAX), + first_seen_ms: created_at_ms, + last_seen_ms: updated_at_ms, + }) +} diff --git a/crates/openshell-server/src/sandbox_index.rs b/crates/openshell-server/src/sandbox_index.rs index 2e515757d2..9ad7c941b5 100644 --- a/crates/openshell-server/src/sandbox_index.rs +++ b/crates/openshell-server/src/sandbox_index.rs @@ -27,18 +27,20 @@ impl SandboxIndex { pub fn update_from_sandbox(&self, sandbox: &Sandbox) { let mut inner = self.inner.write().expect("sandbox index lock poisoned"); - if !sandbox.name.is_empty() { - inner - .sandbox_name_to_id - .insert(sandbox.name.clone(), sandbox.id.clone()); - } + if let Some(metadata) = &sandbox.metadata { + if !metadata.name.is_empty() { + inner + .sandbox_name_to_id + .insert(metadata.name.clone(), metadata.id.clone()); + } - if let Some(status) = sandbox.status.as_ref() - && !status.agent_pod.is_empty() - { - inner - .agent_pod_to_id - .insert(status.agent_pod.clone(), sandbox.id.clone()); + if let Some(status) = sandbox.status.as_ref() + && !status.agent_pod.is_empty() + { + inner + .agent_pod_to_id + .insert(status.agent_pod.clone(), metadata.id.clone()); + } } } diff --git a/crates/openshell-server/src/sandbox_watch.rs b/crates/openshell-server/src/sandbox_watch.rs index 73cc4bf266..ac38eba8db 100644 --- a/crates/openshell-server/src/sandbox_watch.rs +++ b/crates/openshell-server/src/sandbox_watch.rs @@ -59,6 +59,16 @@ impl SandboxWatchBus { } } +/// Helper to translate broadcast lag into a gRPC status. +pub fn broadcast_to_status(err: broadcast::error::RecvError) -> Status { + match err { + broadcast::error::RecvError::Closed => Status::cancelled("stream closed"), + broadcast::error::RecvError::Lagged(n) => { + Status::resource_exhausted(format!("watch stream lagged; dropped {n} messages")) + } + } +} + #[cfg(test)] mod tests { use super::*; @@ -105,13 +115,3 @@ mod tests { bus.remove("nonexistent"); } } - -/// Helper to translate broadcast lag into a gRPC status. -pub fn broadcast_to_status(err: broadcast::error::RecvError) -> Status { - match err { - broadcast::error::RecvError::Closed => Status::cancelled("stream closed"), - broadcast::error::RecvError::Lagged(n) => { - Status::resource_exhausted(format!("watch stream lagged; dropped {n} messages")) - } - } -} diff --git a/crates/openshell-server/src/ssh_tunnel.rs b/crates/openshell-server/src/ssh_tunnel.rs index 536513ccd8..bd317d53fa 100644 --- a/crates/openshell-server/src/ssh_tunnel.rs +++ b/crates/openshell-server/src/ssh_tunnel.rs @@ -6,24 +6,19 @@ use axum::{Router, extract::State, http::Method, response::IntoResponse, routing::any}; use http::StatusCode; use hyper::Request; -use hyper::upgrade::Upgraded; use hyper_util::rt::TokioIo; use openshell_core::proto::{Sandbox, SandboxPhase, SshSession}; use prost::Message; -use std::net::SocketAddr; use std::sync::Arc; use std::time::Duration; -use tokio::io::{AsyncReadExt, AsyncWriteExt}; -use tokio::net::TcpStream; +use tokio::io::AsyncWriteExt; use tracing::{info, warn}; -use uuid::Uuid; use crate::ServerState; -use crate::persistence::{ObjectId, ObjectName, ObjectType, Store}; +use crate::persistence::{ObjectType, Store}; const HEADER_SANDBOX_ID: &str = "x-sandbox-id"; const HEADER_TOKEN: &str = "x-sandbox-token"; -const PREFACE_MAGIC: &str = "NSSH1"; /// Maximum concurrent SSH tunnel connections per session token. const MAX_CONNECTIONS_PER_TOKEN: u32 = 3; @@ -40,6 +35,39 @@ fn redact_token(token: &str) -> String { /// Maximum concurrent SSH tunnel connections per sandbox. const MAX_CONNECTIONS_PER_SANDBOX: u32 = 20; +fn acquire_connection_slots( + token_counts: &std::sync::Mutex>, + sandbox_counts: &std::sync::Mutex>, + token: &str, + sandbox_id: &str, +) -> Result<(), ConnectionLimit> { + { + let mut counts = token_counts.lock().unwrap(); + let count = counts.entry(token.to_string()).or_insert(0); + if *count >= MAX_CONNECTIONS_PER_TOKEN { + return Err(ConnectionLimit::PerToken); + } + *count += 1; + } + + { + let mut counts = sandbox_counts.lock().unwrap(); + let count = counts.entry(sandbox_id.to_string()).or_insert(0); + if *count >= MAX_CONNECTIONS_PER_SANDBOX { + decrement_connection_count(token_counts, token); + return Err(ConnectionLimit::PerSandbox); + } + *count += 1; + } + + Ok(()) +} + +enum ConnectionLimit { + PerToken, + PerSandbox, +} + pub fn router(state: Arc) -> Router { Router::new() .route("/connect/ssh", any(ssh_connect)) @@ -78,10 +106,13 @@ async fn ssh_connect( // Check token expiry (0 means no expiry for backward compatibility). if session.expires_at_ms > 0 { - let now_ms = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_millis() as i64; + let now_ms = i64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_millis(), + ) + .unwrap_or(i64::MAX); if now_ms > session.expires_at_ms { return StatusCode::UNAUTHORIZED.into_response(); } @@ -100,69 +131,82 @@ async fn ssh_connect( return StatusCode::PRECONDITION_FAILED.into_response(); } - let connect_target = match state.compute.resolve_sandbox_endpoint(&sandbox).await { - Ok(crate::compute::ResolvedEndpoint::Ip(ip, port)) => { - ConnectTarget::Ip(SocketAddr::new(ip, port)) - } - Ok(crate::compute::ResolvedEndpoint::Host(host, port)) => ConnectTarget::Host(host, port), - Err(status) if status.code() == tonic::Code::FailedPrecondition => { - return StatusCode::PRECONDITION_FAILED.into_response(); - } - Err(err) => { - warn!(error = %err, "Failed to resolve sandbox endpoint"); - return StatusCode::BAD_GATEWAY.into_response(); - } - }; - // Enforce per-token concurrent connection limit. - { - let mut counts = state.ssh_connections_by_token.lock().unwrap(); - let count = counts.entry(token.clone()).or_insert(0); - if *count >= MAX_CONNECTIONS_PER_TOKEN { - warn!(token = %redact_token(&token), "SSH tunnel: per-token connection limit reached"); - return StatusCode::TOO_MANY_REQUESTS.into_response(); + // Enforce connection caps *before* opening a relay — otherwise denied + // calls churn pending relay slots and wake the supervisor until the relay + // timeout elapses. + if let Err(limit) = acquire_connection_slots( + &state.ssh_connections_by_token, + &state.ssh_connections_by_sandbox, + &token, + &sandbox_id, + ) { + match limit { + ConnectionLimit::PerToken => { + warn!(token = %redact_token(&token), "SSH tunnel: per-token connection limit reached"); + } + ConnectionLimit::PerSandbox => { + warn!(sandbox_id = %sandbox_id, "SSH tunnel: per-sandbox connection limit reached"); + } } - *count += 1; + return StatusCode::TOO_MANY_REQUESTS.into_response(); } - // Enforce per-sandbox concurrent connection limit. + // Open a relay channel through the supervisor session. Use a generous + // 30s session-wait timeout because `/connect/ssh` is typically called + // immediately after `sandbox create`, so we need to cover the supervisor's + // initial TLS + gRPC handshake on a cold-started pod. The old + // direct-connect path tolerated ~34s here for similar reasons. + let (channel_id, relay_rx) = match state + .supervisor_sessions + .open_relay(&sandbox_id, Duration::from_secs(30)) + .await { - let mut counts = state.ssh_connections_by_sandbox.lock().unwrap(); - let count = counts.entry(sandbox_id.clone()).or_insert(0); - if *count >= MAX_CONNECTIONS_PER_SANDBOX { - // Roll back the per-token increment. - let mut token_counts = state.ssh_connections_by_token.lock().unwrap(); - if let Some(c) = token_counts.get_mut(&token) { - *c = c.saturating_sub(1); - if *c == 0 { - token_counts.remove(&token); - } - } - warn!(sandbox_id = %sandbox_id, "SSH tunnel: per-sandbox connection limit reached"); - return StatusCode::TOO_MANY_REQUESTS.into_response(); + Ok(pair) => pair, + Err(status) => { + warn!(sandbox_id = %sandbox_id, error = %status.message(), "SSH tunnel: supervisor session not available"); + decrement_connection_count(&state.ssh_connections_by_token, &token); + decrement_connection_count(&state.ssh_connections_by_sandbox, &sandbox_id); + return StatusCode::BAD_GATEWAY.into_response(); } - *count += 1; - } + }; - let handshake_secret = state.config.ssh_handshake_secret.clone(); let sandbox_id_clone = sandbox_id.clone(); let token_clone = token.clone(); let state_clone = state.clone(); let upgrade = hyper::upgrade::on(req); tokio::spawn(async move { - match upgrade.await { - Ok(mut upgraded) => { - if let Err(err) = handle_tunnel( - &mut upgraded, - connect_target, - &token_clone, - &handshake_secret, + // Wait for the supervisor to open its `RelayStream` and deliver the + // bridge half of the relay. + let mut relay = match tokio::time::timeout(Duration::from_secs(10), relay_rx).await { + Ok(Ok(stream)) => stream, + Ok(Err(_)) => { + warn!(sandbox_id = %sandbox_id_clone, channel_id = %channel_id, "SSH tunnel: relay channel dropped"); + decrement_connection_count(&state_clone.ssh_connections_by_token, &token_clone); + decrement_connection_count( + &state_clone.ssh_connections_by_sandbox, &sandbox_id_clone, - ) - .await - { - warn!(error = %err, "SSH tunnel failure"); - } + ); + return; + } + Err(_) => { + warn!(sandbox_id = %sandbox_id_clone, channel_id = %channel_id, "SSH tunnel: relay open timed out"); + decrement_connection_count(&state_clone.ssh_connections_by_token, &token_clone); + decrement_connection_count( + &state_clone.ssh_connections_by_sandbox, + &sandbox_id_clone, + ); + return; + } + }; + + info!(sandbox_id = %sandbox_id_clone, channel_id = %channel_id, "SSH tunnel: relay established, bridging client"); + + match upgrade.await { + Ok(upgraded) => { + let mut upgraded = TokioIo::new(upgraded); + let _ = tokio::io::copy_bidirectional(&mut upgraded, &mut relay).await; + let _ = AsyncWriteExt::shutdown(&mut upgraded).await; } Err(err) => { warn!(error = %err, "SSH upgrade failed"); @@ -177,90 +221,6 @@ async fn ssh_connect( StatusCode::OK.into_response() } -async fn handle_tunnel( - upgraded: &mut Upgraded, - target: ConnectTarget, - token: &str, - secret: &str, - sandbox_id: &str, -) -> Result<(), Box> { - // The sandbox pod may not be network-reachable immediately after the CRD - // reports Ready (DNS propagation, pod IP assignment, SSH server startup). - // Retry the TCP connection with exponential backoff. - let mut upstream = None; - let mut last_err = None; - let delays = [ - Duration::from_millis(100), - Duration::from_millis(250), - Duration::from_millis(500), - Duration::from_secs(1), - Duration::from_secs(2), - Duration::from_secs(5), - Duration::from_secs(10), - Duration::from_secs(15), - ]; - let target_desc = match &target { - ConnectTarget::Ip(addr) => format!("{addr}"), - ConnectTarget::Host(host, port) => format!("{host}:{port}"), - }; - info!(sandbox_id = %sandbox_id, target = %target_desc, "SSH tunnel: connecting to sandbox"); - for (attempt, delay) in std::iter::once(&Duration::ZERO) - .chain(delays.iter()) - .enumerate() - { - if !delay.is_zero() { - info!(sandbox_id = %sandbox_id, attempt = attempt + 1, delay_ms = delay.as_millis() as u64, "SSH tunnel: retrying TCP connect"); - tokio::time::sleep(*delay).await; - } - let result = match &target { - ConnectTarget::Ip(addr) => TcpStream::connect(addr).await, - ConnectTarget::Host(host, port) => TcpStream::connect((host.as_str(), *port)).await, - }; - match result { - Ok(stream) => { - info!( - sandbox_id = %sandbox_id, - attempts = attempt + 1, - "SSH tunnel: TCP connected to sandbox" - ); - upstream = Some(stream); - break; - } - Err(err) => { - info!(sandbox_id = %sandbox_id, attempt = attempt + 1, error = %err, "SSH tunnel: TCP connect failed"); - last_err = Some(err); - } - } - } - let mut upstream = upstream.ok_or_else(|| { - let err = last_err.unwrap(); - format!("failed to connect to sandbox after retries: {err}") - })?; - upstream.set_nodelay(true)?; - info!(sandbox_id = %sandbox_id, "SSH tunnel: sending NSSH1 handshake preface"); - let preface = build_preface(token, secret)?; - upstream.write_all(preface.as_bytes()).await?; - - info!(sandbox_id = %sandbox_id, "SSH tunnel: waiting for handshake response"); - let mut response = String::new(); - read_line(&mut upstream, &mut response).await?; - info!(sandbox_id = %sandbox_id, response = %response.trim(), "SSH tunnel: handshake response received"); - if response.trim() != "OK" { - return Err("sandbox handshake rejected".into()); - } - - info!(sandbox_id = %sandbox_id, "SSH tunnel established"); - let mut upgraded = TokioIo::new(upgraded); - // Discard the result entirely – connection-close errors are expected when - // the SSH session ends and do not represent a failure worth propagating. - let _ = tokio::io::copy_bidirectional(&mut upgraded, &mut upstream).await; - // Gracefully shut down the write-half of the upgraded connection so the - // client receives a clean EOF instead of a TCP RST. This gives SSH time - // to read any remaining protocol data (e.g. exit-status) from its buffer. - let _ = AsyncWriteExt::shutdown(&mut upgraded).await; - Ok(()) -} - fn header_value(headers: &http::HeaderMap, name: &str) -> Result { let value = headers .get(name) @@ -275,81 +235,12 @@ fn header_value(headers: &http::HeaderMap, name: &str) -> Result Result> { - let timestamp = i64::try_from( - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map_err(|_| "time error")? - .as_secs(), - ) - .map_err(|_| "time error")?; - let nonce = Uuid::new_v4().to_string(); - let payload = format!("{token}|{timestamp}|{nonce}"); - let signature = hmac_sha256(secret.as_bytes(), payload.as_bytes()); - Ok(format!( - "{PREFACE_MAGIC} {token} {timestamp} {nonce} {signature}\n" - )) -} - -async fn read_line( - stream: &mut TcpStream, - buf: &mut String, -) -> Result<(), Box> { - let mut bytes = Vec::new(); - loop { - let mut byte = [0u8; 1]; - let n = stream.read(&mut byte).await?; - if n == 0 { - break; - } - if byte[0] == b'\n' { - break; - } - bytes.push(byte[0]); - if bytes.len() > 1024 { - break; - } - } - *buf = String::from_utf8_lossy(&bytes).to_string(); - Ok(()) -} - -fn hmac_sha256(key: &[u8], data: &[u8]) -> String { - use hmac::{Hmac, Mac}; - use sha2::Sha256; - - let mut mac = Hmac::::new_from_slice(key).expect("hmac key"); - mac.update(data); - let result = mac.finalize().into_bytes(); - hex::encode(result) -} - impl ObjectType for SshSession { fn object_type() -> &'static str { "ssh_session" } } -impl ObjectId for SshSession { - fn object_id(&self) -> &str { - &self.id - } -} - -impl ObjectName for SshSession { - fn object_name(&self) -> &str { - &self.name - } -} - -enum ConnectTarget { - Ip(SocketAddr), - Host(String, u16), -} - /// Decrement a connection count entry, removing it if it reaches zero. fn decrement_connection_count( counts: &std::sync::Mutex>, @@ -380,10 +271,13 @@ pub fn spawn_session_reaper(store: Arc, interval: Duration) { } async fn reap_expired_sessions(store: &Store) -> Result<(), String> { - let now_ms = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_millis() as i64; + let now_ms = i64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_millis(), + ) + .unwrap_or(i64::MAX); let records = store .list(SshSession::object_type(), 1000, 0) @@ -404,8 +298,12 @@ async fn reap_expired_sessions(store: &Store) -> Result<(), String> { || session.revoked; if should_delete { - if let Err(e) = store.delete(SshSession::object_type(), &session.id).await { - warn!(session_id = %session.id, error = %e, "Failed to reap SSH session"); + use openshell_core::ObjectId; + if let Err(e) = store + .delete(SshSession::object_type(), session.object_id()) + .await + { + warn!(session_id = %session.object_id(), error = %e, "Failed to reap SSH session"); } else { reaped += 1; } @@ -427,21 +325,27 @@ mod tests { fn make_session(id: &str, sandbox_id: &str, expires_at_ms: i64, revoked: bool) -> SshSession { SshSession { - id: id.to_string(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: id.to_string(), + name: format!("session-{id}"), + created_at_ms: 1000, + labels: HashMap::new(), + }), sandbox_id: sandbox_id.to_string(), token: id.to_string(), - created_at_ms: 1000, - revoked, - name: format!("session-{id}"), expires_at_ms, + revoked, } } fn now_ms() -> i64 { - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap() - .as_millis() as i64 + i64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_millis(), + ) + .unwrap_or(i64::MAX) } // ---- Connection limit tests ---- @@ -491,6 +395,36 @@ mod tests { assert!(current >= MAX_CONNECTIONS_PER_SANDBOX); } + #[test] + fn acquire_connection_slots_rejects_per_token_limit_without_touching_sandbox() { + let token_counts: Mutex> = Mutex::new(HashMap::new()); + let sandbox_counts: Mutex> = Mutex::new(HashMap::new()); + token_counts + .lock() + .unwrap() + .insert("tok1".to_string(), MAX_CONNECTIONS_PER_TOKEN); + + let result = acquire_connection_slots(&token_counts, &sandbox_counts, "tok1", "sbx1"); + + assert!(matches!(result, Err(ConnectionLimit::PerToken))); + assert!(sandbox_counts.lock().unwrap().is_empty()); + } + + #[test] + fn acquire_connection_slots_rolls_back_token_increment_on_sandbox_limit() { + let token_counts: Mutex> = Mutex::new(HashMap::new()); + let sandbox_counts: Mutex> = Mutex::new(HashMap::new()); + sandbox_counts + .lock() + .unwrap() + .insert("sbx1".to_string(), MAX_CONNECTIONS_PER_SANDBOX); + + let result = acquire_connection_slots(&token_counts, &sandbox_counts, "tok1", "sbx1"); + + assert!(matches!(result, Err(ConnectionLimit::PerSandbox))); + assert!(token_counts.lock().unwrap().is_empty()); + } + // ---- Session reaper tests ---- #[tokio::test] diff --git a/crates/openshell-server/src/supervisor_session.rs b/crates/openshell-server/src/supervisor_session.rs new file mode 100644 index 0000000000..94c352ba53 --- /dev/null +++ b/crates/openshell-server/src/supervisor_session.rs @@ -0,0 +1,1316 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::collections::HashMap; +use std::pin::Pin; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use tokio::sync::{mpsc, oneshot}; +use tokio_stream::wrappers::ReceiverStream; +use tonic::{Request, Response, Status}; +use tracing::{info, warn}; +use uuid::Uuid; + +use openshell_core::proto::{ + GatewayMessage, RelayFrame, RelayInit, RelayOpen, Sandbox, SessionAccepted, SupervisorMessage, + gateway_message, supervisor_message, +}; + +use crate::ServerState; + +const HEARTBEAT_INTERVAL_SECS: u32 = 15; +const RELAY_PENDING_TIMEOUT: Duration = Duration::from_secs(10); +/// Initial backoff between session-availability polls in `wait_for_session`. +const SESSION_WAIT_INITIAL_BACKOFF: Duration = Duration::from_millis(100); +/// Maximum backoff between session-availability polls in `wait_for_session`. +const SESSION_WAIT_MAX_BACKOFF: Duration = Duration::from_secs(2); +/// Upper bound on unclaimed relay channels across all sandboxes. Caps the +/// memory a misbehaving caller can pin by calling `open_relay` repeatedly +/// while the supervisor never claims (or isn't responding). Sized generously +/// so normal bursts pass through; exceeding it returns `ResourceExhausted`. +const MAX_PENDING_RELAYS: usize = 256; +/// Upper bound on concurrent unclaimed relay channels for a single sandbox. +/// Enforces the same shape per sandbox so one misbehaving sandbox can't +/// consume the entire global budget. Sits above the SSH-tunnel per-sandbox +/// cap (20) so tunnel-specific limits still fire first for that caller. +const MAX_PENDING_RELAYS_PER_SANDBOX: usize = 32; + +// --------------------------------------------------------------------------- +// Session registry +// --------------------------------------------------------------------------- + +/// A live supervisor session handle. +struct LiveSession { + #[allow(dead_code)] + sandbox_id: String, + /// Uniquely identifies this session instance. Used by cleanup to avoid + /// removing a session that has since been superseded by a reconnect. + session_id: String, + tx: mpsc::Sender, + /// Fires when this session is superseded by a reconnect so the old session + /// task can exit promptly — dropping its own `tx` clone and closing the + /// outbound stream. Without this, a concurrent `open_relay` that grabbed + /// the old session's `tx` just before supersede could still enqueue a + /// `RelayOpen` onto the stale stream and sit until the relay timeout. + shutdown: oneshot::Sender<()>, + #[allow(dead_code)] + connected_at: Instant, +} + +/// Holds a oneshot sender that will deliver the upgraded relay stream. +type RelayStreamSender = oneshot::Sender; + +impl openshell_driver_docker::SupervisorReadiness for SupervisorSessionRegistry { + fn is_supervisor_connected(&self, sandbox_id: &str) -> bool { + Self::is_connected(self, sandbox_id) + } +} + +/// Registry of active supervisor sessions and pending relay channels. +#[derive(Default)] +pub struct SupervisorSessionRegistry { + /// `sandbox_id` -> live session handle. + sessions: Mutex>, + /// `channel_id` -> oneshot sender for the reverse CONNECT stream. + pending_relays: Mutex>, +} + +struct PendingRelay { + sender: RelayStreamSender, + sandbox_id: String, + created_at: Instant, +} + +impl std::fmt::Debug for SupervisorSessionRegistry { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let session_count = self.sessions.lock().unwrap().len(); + let pending_count = self.pending_relays.lock().unwrap().len(); + f.debug_struct("SupervisorSessionRegistry") + .field("sessions", &session_count) + .field("pending_relays", &pending_count) + .finish() + } +} + +impl SupervisorSessionRegistry { + pub fn new() -> Self { + Self::default() + } + + /// Register a live supervisor session for the given sandbox. + /// + /// If a previous session exists for the same sandbox, its shutdown signal + /// is fired so the old session task exits promptly. Returns `true` iff a + /// previous session was superseded. + pub fn register( + &self, + sandbox_id: String, + session_id: String, + tx: mpsc::Sender, + shutdown: oneshot::Sender<()>, + ) -> bool { + let mut sessions = self.sessions.lock().unwrap(); + let previous = sessions.remove(&sandbox_id); + sessions.insert( + sandbox_id.clone(), + LiveSession { + sandbox_id, + session_id, + tx, + shutdown, + connected_at: Instant::now(), + }, + ); + match previous { + Some(prev) => { + // Best-effort — the old task may have already exited. + let _ = prev.shutdown.send(()); + true + } + None => false, + } + } + + /// Report whether a live supervisor session is registered for a sandbox. + /// + /// Used by compute drivers that need to surface "supervisor relay ready" + /// through the Ready condition without polling the sandbox runtime. + pub fn is_connected(&self, sandbox_id: &str) -> bool { + self.sessions.lock().unwrap().contains_key(sandbox_id) + } + + /// Remove the session for a sandbox. + fn remove(&self, sandbox_id: &str) { + self.sessions.lock().unwrap().remove(sandbox_id); + } + + /// Remove the session only if its `session_id` matches the one we are + /// cleaning up. Returns `true` if the entry was removed. + /// + /// This guards against the supersede race: an old session's task may + /// finish long after a new session has taken its place. The old task's + /// cleanup must not evict the new registration. + fn remove_if_current(&self, sandbox_id: &str, session_id: &str) -> bool { + let mut sessions = self.sessions.lock().unwrap(); + let is_current = sessions + .get(sandbox_id) + .is_some_and(|s| s.session_id == session_id); + if is_current { + sessions.remove(sandbox_id); + } + is_current + } + + /// Look up the sender for a supervisor session, waiting up to `timeout` + /// for it to appear if absent. + /// + /// Uses exponential backoff (100ms → 2s) while polling the sessions map. + async fn wait_for_session( + &self, + sandbox_id: &str, + timeout: Duration, + ) -> Result, Status> { + let deadline = Instant::now() + timeout; + let mut backoff = SESSION_WAIT_INITIAL_BACKOFF; + + loop { + if let Some(tx) = self.lookup_session(sandbox_id) { + return Ok(tx); + } + if Instant::now() + backoff > deadline { + return Err(Status::unavailable("supervisor session not connected")); + } + tokio::time::sleep(backoff).await; + backoff = (backoff * 2).min(SESSION_WAIT_MAX_BACKOFF); + } + } + + fn lookup_session(&self, sandbox_id: &str) -> Option> { + self.sessions + .lock() + .unwrap() + .get(sandbox_id) + .map(|s| s.tx.clone()) + } + + pub fn has_session(&self, sandbox_id: &str) -> bool { + self.sessions.lock().unwrap().contains_key(sandbox_id) + } + + fn pending_channel_ids(&self, sandbox_id: &str) -> Vec { + self.pending_relays + .lock() + .unwrap() + .iter() + .filter(|(_, pending)| pending.sandbox_id == sandbox_id) + .map(|(channel_id, _)| channel_id.clone()) + .collect() + } + + /// Open a relay channel and return a receiver for the supervisor-side + /// stream. + /// + /// Sends `RelayOpen` over the supervisor's gRPC session and returns a + /// oneshot receiver that resolves once the supervisor opens its reverse + /// HTTP CONNECT to `/relay/{channel_id}`. + /// + /// If the session is not currently registered, this method waits up to + /// `session_wait_timeout` for it to appear. A session may be temporarily + /// absent for several reasons — all of which look identical from here: + /// + /// - startup race: the sandbox just reported Ready but the supervisor's + /// `ConnectSupervisor` gRPC handshake hasn't completed yet + /// - transient disconnect: the session was up but got dropped (network + /// blip, gateway restart, supervisor restart) and the supervisor is + /// in its reconnect backoff loop + /// + /// Callers pick the timeout based on how much patience the caller needs. + /// A first `sandbox connect` right after `sandbox create` may need to + /// wait for the supervisor's initial TLS + gRPC handshake (tens of + /// seconds on a slow cluster), while mid-lifetime calls typically just + /// need to cover a short reconnect window. + pub async fn open_relay( + &self, + sandbox_id: &str, + session_wait_timeout: Duration, + ) -> Result<(String, oneshot::Receiver), Status> { + let tx = self + .wait_for_session(sandbox_id, session_wait_timeout) + .await?; + + let channel_id = Uuid::new_v4().to_string(); + + // Register the pending relay before sending RelayOpen to avoid a race. + // Both caps are checked and the insert happens under a single lock hold + // so two concurrent calls can't both observe "under the cap" and then + // both insert past it. + let (relay_tx, relay_rx) = oneshot::channel(); + { + let mut pending = self.pending_relays.lock().unwrap(); + if pending.len() >= MAX_PENDING_RELAYS { + return Err(Status::resource_exhausted(format!( + "gateway relay capacity reached ({MAX_PENDING_RELAYS} in flight)" + ))); + } + let per_sandbox = pending + .values() + .filter(|p| p.sandbox_id == sandbox_id) + .count(); + if per_sandbox >= MAX_PENDING_RELAYS_PER_SANDBOX { + return Err(Status::resource_exhausted(format!( + "per-sandbox relay limit reached ({MAX_PENDING_RELAYS_PER_SANDBOX} in flight for {sandbox_id})" + ))); + } + pending.insert( + channel_id.clone(), + PendingRelay { + sender: relay_tx, + sandbox_id: sandbox_id.to_string(), + created_at: Instant::now(), + }, + ); + } + + let msg = GatewayMessage { + payload: Some(gateway_message::Payload::RelayOpen(RelayOpen { + channel_id: channel_id.clone(), + })), + }; + + if tx.send(msg).await.is_err() { + // Session dropped between our lookup and send. + self.pending_relays.lock().unwrap().remove(&channel_id); + return Err(Status::unavailable("supervisor session disconnected")); + } + + Ok((channel_id, relay_rx)) + } + + /// Claim a pending relay channel. Called by the `/relay/{channel_id}` HTTP handler + /// when the supervisor's reverse CONNECT arrives. + /// + /// Returns the `DuplexStream` half that the supervisor side should read/write. + // `tonic::Status` is large but is the API surface of gRPC handlers. + #[allow(clippy::result_large_err)] + pub fn claim_relay(&self, channel_id: &str) -> Result { + let pending = { + let mut map = self.pending_relays.lock().unwrap(); + map.remove(channel_id) + .ok_or_else(|| Status::not_found("unknown or expired relay channel"))? + }; + + if pending.created_at.elapsed() > RELAY_PENDING_TIMEOUT { + return Err(Status::deadline_exceeded("relay channel timed out")); + } + + // Create a duplex stream pair: one end for the gateway bridge, one for + // the supervisor HTTP CONNECT handler. + let (gateway_stream, supervisor_stream) = tokio::io::duplex(64 * 1024); + + // Send the gateway-side stream to the waiter (ssh_tunnel or exec handler). + if pending.sender.send(gateway_stream).is_err() { + return Err(Status::internal("relay requester dropped")); + } + + Ok(supervisor_stream) + } + + /// Remove all pending relays that have exceeded the timeout. + pub fn reap_expired_relays(&self) { + let mut map = self.pending_relays.lock().unwrap(); + map.retain(|_, pending| pending.created_at.elapsed() <= RELAY_PENDING_TIMEOUT); + } + + /// Clean up all state for a sandbox (session + pending relays). + pub fn cleanup_sandbox(&self, sandbox_id: &str) { + self.remove(sandbox_id); + } + + pub async fn replay_pending_relays(&self, sandbox_id: &str, tx: &mpsc::Sender) { + for channel_id in self.pending_channel_ids(sandbox_id) { + let msg = GatewayMessage { + payload: Some(gateway_message::Payload::RelayOpen(RelayOpen { + channel_id: channel_id.clone(), + })), + }; + if tx.send(msg).await.is_err() { + warn!(sandbox_id = %sandbox_id, channel_id = %channel_id, "supervisor session: failed to replay pending relay to superseding session"); + break; + } + } + } +} + +/// Spawn a background task that periodically reaps expired pending relay +/// entries. +/// +/// Pending entries are normally consumed either when the supervisor opens its +/// reverse CONNECT (via `claim_relay`) or by the gateway-side waiter timing +/// out. If neither happens — e.g., the supervisor crashed after acknowledging +/// `RelayOpen` but before initiating `RelayStream` — the entry would otherwise +/// sit in the map indefinitely. This sweeper bounds that leak. +pub fn spawn_relay_reaper(state: Arc, interval: Duration) { + tokio::spawn(async move { + loop { + tokio::time::sleep(interval).await; + state.supervisor_sessions.reap_expired_relays(); + } + }); +} + +async fn require_persisted_sandbox( + store: &Arc, + sandbox_id: &str, +) -> Result<(), Status> { + let sandbox = store + .get_message::(sandbox_id) + .await + .map_err(|err| Status::internal(format!("failed to load sandbox: {err}")))?; + + if sandbox.is_none() { + return Err(Status::not_found("sandbox not found")); + } + + Ok(()) +} + +// --------------------------------------------------------------------------- +// RelayStream gRPC handler +// --------------------------------------------------------------------------- + +/// Size of chunks read from the gateway-side `DuplexStream` when forwarding +/// bytes back to the supervisor over the gRPC response stream. +const RELAY_STREAM_CHUNK_SIZE: usize = 16 * 1024; + +/// Handle a `RelayStream` RPC from a supervisor. +/// +/// The first inbound `RelayFrame` must carry a `RelayInit` identifying the +/// pending relay; subsequent frames carry raw bytes forward to the +/// gateway-side waiter. Bytes flowing the other way are chunked and sent as +/// `RelayFrame::data` messages back over the response stream. +pub async fn handle_relay_stream( + registry: &SupervisorSessionRegistry, + request: Request>, +) -> Result< + Response< + Pin> + Send + 'static>>, + >, + Status, +> { + let mut inbound = request.into_inner(); + + // First frame must identify the channel. + let first = inbound + .message() + .await? + .ok_or_else(|| Status::invalid_argument("empty RelayStream"))?; + let channel_id = match first.payload { + Some(openshell_core::proto::relay_frame::Payload::Init(RelayInit { channel_id })) + if !channel_id.is_empty() => + { + channel_id + } + _ => { + return Err(Status::invalid_argument( + "first RelayFrame must be init with non-empty channel_id", + )); + } + }; + + // Claim the pending relay. Consumes the entry — it cannot be reused. + let supervisor_side = registry.claim_relay(&channel_id)?; + info!(channel_id = %channel_id, "relay stream: claimed pending relay, bridging"); + + let (mut read_half, mut write_half) = tokio::io::split(supervisor_side); + + // Supervisor → gateway: drain `inbound` and write to the DuplexStream. + let channel_id_in = channel_id.clone(); + tokio::spawn(async move { + loop { + match inbound.message().await { + Ok(Some(frame)) => { + let Some(openshell_core::proto::relay_frame::Payload::Data(data)) = + frame.payload + else { + warn!(channel_id = %channel_id_in, "relay stream: received non-data frame after init"); + break; + }; + if data.is_empty() { + continue; + } + if let Err(e) = + tokio::io::AsyncWriteExt::write_all(&mut write_half, &data).await + { + warn!(channel_id = %channel_id_in, error = %e, "relay stream: write to duplex failed"); + break; + } + } + Ok(None) => break, + Err(e) => { + warn!(channel_id = %channel_id_in, error = %e, "relay stream: inbound errored"); + break; + } + } + } + // Best-effort half-close on the write side so the reader sees EOF. + let _ = tokio::io::AsyncWriteExt::shutdown(&mut write_half).await; + }); + + // Gateway → supervisor: read the DuplexStream and emit RelayFrame::data messages. + let (out_tx, out_rx) = mpsc::channel::>(16); + let channel_id_out = channel_id; + tokio::spawn(async move { + let mut buf = vec![0u8; RELAY_STREAM_CHUNK_SIZE]; + loop { + match tokio::io::AsyncReadExt::read(&mut read_half, &mut buf).await { + Ok(0) => break, + Ok(n) => { + let chunk = RelayFrame { + payload: Some(openshell_core::proto::relay_frame::Payload::Data( + buf[..n].to_vec(), + )), + }; + if out_tx.send(Ok(chunk)).await.is_err() { + break; + } + } + Err(e) => { + warn!(channel_id = %channel_id_out, error = %e, "relay stream: read from duplex failed"); + break; + } + } + } + }); + + let stream = ReceiverStream::new(out_rx); + let stream: Pin< + Box> + Send + 'static>, + > = Box::pin(stream); + Ok(Response::new(stream)) +} + +// --------------------------------------------------------------------------- +// ConnectSupervisor gRPC handler +// --------------------------------------------------------------------------- + +pub async fn handle_connect_supervisor( + state: &Arc, + request: Request>, +) -> Result< + Response< + Pin> + Send + 'static>>, + >, + Status, +> { + let mut inbound = request.into_inner(); + + // Step 1: Wait for SupervisorHello. + let hello = match inbound.message().await? { + Some(msg) => match msg.payload { + Some(supervisor_message::Payload::Hello(hello)) => hello, + _ => return Err(Status::invalid_argument("expected SupervisorHello")), + }, + None => return Err(Status::invalid_argument("stream closed before hello")), + }; + + let sandbox_id = hello.sandbox_id.clone(); + if sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + require_persisted_sandbox(&state.store, &sandbox_id).await?; + + let session_id = Uuid::new_v4().to_string(); + info!( + sandbox_id = %sandbox_id, + session_id = %session_id, + instance_id = %hello.instance_id, + "supervisor session: accepted" + ); + + // Step 2: Create the outbound channel and register the session. + let (tx, rx) = mpsc::channel::(64); + let (shutdown_tx, shutdown_rx) = oneshot::channel::<()>(); + let superseded = state.supervisor_sessions.register( + sandbox_id.clone(), + session_id.clone(), + tx.clone(), + shutdown_tx, + ); + if superseded { + info!( + sandbox_id = %sandbox_id, + session_id = %session_id, + "supervisor session: superseded previous session" + ); + } + + // Step 3: Send SessionAccepted. + let accepted = GatewayMessage { + payload: Some(gateway_message::Payload::SessionAccepted(SessionAccepted { + session_id: session_id.clone(), + heartbeat_interval_secs: HEARTBEAT_INTERVAL_SECS, + })), + }; + if tx.send(accepted).await.is_err() { + // Only evict ourselves — a faster reconnect may already have + // superseded this registration. + state + .supervisor_sessions + .remove_if_current(&sandbox_id, &session_id); + return Err(Status::internal("failed to send session accepted")); + } + + if superseded { + state + .supervisor_sessions + .replay_pending_relays(&sandbox_id, &tx) + .await; + } + + if let Err(err) = state + .compute + .supervisor_session_connected(&sandbox_id) + .await + { + warn!( + sandbox_id = %sandbox_id, + session_id = %session_id, + error = %err, + "supervisor session: failed to mark sandbox ready" + ); + } + + // Step 4: Spawn the session loop that reads inbound messages. + let state_clone = Arc::clone(state); + let sandbox_id_clone = sandbox_id.clone(); + tokio::spawn(async move { + run_session_loop( + &state_clone, + &sandbox_id_clone, + &session_id, + &tx, + &mut inbound, + shutdown_rx, + ) + .await; + let still_ours = state_clone + .supervisor_sessions + .remove_if_current(&sandbox_id_clone, &session_id); + if still_ours { + info!(sandbox_id = %sandbox_id_clone, session_id = %session_id, "supervisor session: ended"); + if let Err(err) = state_clone + .compute + .supervisor_session_disconnected(&sandbox_id_clone) + .await + { + warn!( + sandbox_id = %sandbox_id_clone, + session_id = %session_id, + error = %err, + "supervisor session: failed to mark sandbox disconnected" + ); + } + } else { + info!(sandbox_id = %sandbox_id_clone, session_id = %session_id, "supervisor session: ended (already superseded)"); + } + }); + + // Return the outbound stream. + let stream = ReceiverStream::new(rx); + let stream: Pin< + Box> + Send + 'static>, + > = Box::pin(tokio_stream::StreamExt::map(stream, Ok)); + + Ok(Response::new(stream)) +} + +async fn run_session_loop( + _state: &Arc, + sandbox_id: &str, + session_id: &str, + tx: &mpsc::Sender, + inbound: &mut tonic::Streaming, + mut shutdown_rx: oneshot::Receiver<()>, +) { + let heartbeat_interval = Duration::from_secs(u64::from(HEARTBEAT_INTERVAL_SECS)); + let mut heartbeat_timer = tokio::time::interval(heartbeat_interval); + // Skip the first immediate tick. + heartbeat_timer.tick().await; + + loop { + tokio::select! { + _ = &mut shutdown_rx => { + info!(sandbox_id = %sandbox_id, session_id = %session_id, "supervisor session: superseded by reconnect, shutting down"); + break; + } + msg = inbound.message() => { + match msg { + Ok(Some(msg)) => { + handle_supervisor_message(sandbox_id, session_id, msg); + } + Ok(None) => { + info!(sandbox_id = %sandbox_id, session_id = %session_id, "supervisor session: stream closed by supervisor"); + break; + } + Err(e) => { + warn!(sandbox_id = %sandbox_id, session_id = %session_id, error = %e, "supervisor session: stream error"); + break; + } + } + } + _ = heartbeat_timer.tick() => { + let hb = GatewayMessage { + payload: Some(gateway_message::Payload::Heartbeat( + openshell_core::proto::GatewayHeartbeat {}, + )), + }; + if tx.send(hb).await.is_err() { + info!(sandbox_id = %sandbox_id, session_id = %session_id, "supervisor session: outbound channel closed"); + break; + } + } + } + } +} + +fn handle_supervisor_message(sandbox_id: &str, session_id: &str, msg: SupervisorMessage) { + match msg.payload { + Some(supervisor_message::Payload::Heartbeat(_)) => { + // Heartbeat received — nothing to do for now. + } + Some(supervisor_message::Payload::RelayOpenResult(result)) => { + if result.success { + info!( + sandbox_id = %sandbox_id, + session_id = %session_id, + channel_id = %result.channel_id, + "supervisor session: relay opened successfully" + ); + } else { + warn!( + sandbox_id = %sandbox_id, + session_id = %session_id, + channel_id = %result.channel_id, + error = %result.error, + "supervisor session: relay open failed" + ); + } + } + Some(supervisor_message::Payload::RelayClose(close)) => { + info!( + sandbox_id = %sandbox_id, + session_id = %session_id, + channel_id = %close.channel_id, + reason = %close.reason, + "supervisor session: relay closed by supervisor" + ); + } + _ => { + warn!( + sandbox_id = %sandbox_id, + session_id = %session_id, + "supervisor session: unexpected message type" + ); + } + } +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +#[cfg(test)] +mod tests { + use super::*; + use crate::persistence::Store; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + /// Returns a shutdown sender with its receiver immediately dropped. Tests + /// that don't observe the shutdown signal can use this to satisfy the + /// `register` signature without the receiver noise. + fn make_shutdown() -> oneshot::Sender<()> { + oneshot::channel::<()>().0 + } + + fn sandbox_record(id: &str, name: &str) -> Sandbox { + Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: id.to_string(), + name: name.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + }), + ..Default::default() + } + } + + // ---- registry: register / remove ---- + + #[test] + fn registry_register_and_lookup() { + let registry = SupervisorSessionRegistry::new(); + let (tx, _rx) = mpsc::channel(1); + + assert!(!registry.register( + "sandbox-1".to_string(), + "s1".to_string(), + tx, + make_shutdown(), + )); + + let sessions = registry.sessions.lock().unwrap(); + assert!(sessions.contains_key("sandbox-1")); + } + + #[test] + fn registry_supersedes_previous_session() { + let registry = SupervisorSessionRegistry::new(); + let (tx1, _rx1) = mpsc::channel(1); + let (tx2, _rx2) = mpsc::channel(1); + + assert!(!registry.register( + "sandbox-1".to_string(), + "s1".to_string(), + tx1, + make_shutdown(), + )); + assert!(registry.register( + "sandbox-1".to_string(), + "s2".to_string(), + tx2, + make_shutdown(), + )); + } + + #[test] + fn registry_remove() { + let registry = SupervisorSessionRegistry::new(); + let (tx, _rx) = mpsc::channel(1); + registry.register( + "sandbox-1".to_string(), + "s1".to_string(), + tx, + make_shutdown(), + ); + + registry.remove("sandbox-1"); + let sessions = registry.sessions.lock().unwrap(); + assert!(!sessions.contains_key("sandbox-1")); + } + + #[test] + fn remove_if_current_removes_matching_session() { + let registry = SupervisorSessionRegistry::new(); + let (tx, _rx) = mpsc::channel(1); + registry.register("sbx".to_string(), "s1".to_string(), tx, make_shutdown()); + + assert!(registry.remove_if_current("sbx", "s1")); + assert!(!registry.sessions.lock().unwrap().contains_key("sbx")); + } + + #[test] + fn remove_if_current_ignores_stale_session_id() { + let registry = SupervisorSessionRegistry::new(); + let (tx_old, _rx_old) = mpsc::channel(1); + let (tx_new, _rx_new) = mpsc::channel(1); + + // Old session registers, then is superseded by a new session. + registry.register( + "sbx".to_string(), + "s-old".to_string(), + tx_old, + make_shutdown(), + ); + registry.register( + "sbx".to_string(), + "s-new".to_string(), + tx_new, + make_shutdown(), + ); + + // Cleanup from the old session task runs late. It must NOT evict the + // newly registered session. + assert!(!registry.remove_if_current("sbx", "s-old")); + let sessions = registry.sessions.lock().unwrap(); + assert!( + sessions.contains_key("sbx"), + "new session must still be registered" + ); + assert_eq!(sessions.get("sbx").unwrap().session_id, "s-new"); + } + + #[test] + fn remove_if_current_unknown_sandbox_is_noop() { + let registry = SupervisorSessionRegistry::new(); + assert!(!registry.remove_if_current("sbx-does-not-exist", "s1")); + } + + // ---- open_relay: happy path and wait semantics ---- + + #[tokio::test] + async fn open_relay_sends_relay_open_to_registered_session() { + let registry = SupervisorSessionRegistry::new(); + let (tx, mut rx) = mpsc::channel(4); + registry.register("sbx".to_string(), "s1".to_string(), tx, make_shutdown()); + + let (channel_id, _relay_rx) = registry + .open_relay("sbx", Duration::from_secs(1)) + .await + .expect("open_relay should succeed when session is live"); + + let msg = rx.recv().await.expect("relay open should be delivered"); + match msg.payload { + Some(gateway_message::Payload::RelayOpen(open)) => { + assert_eq!(open.channel_id, channel_id); + } + other => panic!("expected RelayOpen, got {other:?}"), + } + } + + #[tokio::test] + async fn open_relay_times_out_without_session() { + let registry = SupervisorSessionRegistry::new(); + let err = registry + .open_relay("missing", Duration::from_millis(50)) + .await + .expect_err("open_relay should time out"); + assert_eq!(err.code(), tonic::Code::Unavailable); + } + + #[tokio::test] + async fn open_relay_waits_for_session_to_appear() { + let registry = Arc::new(SupervisorSessionRegistry::new()); + let registry_for_register = Arc::clone(®istry); + + // Register the session after a small delay, shorter than the wait. + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(200)).await; + let (tx, mut rx) = mpsc::channel::(4); + // Keep the receiver alive so the send in open_relay succeeds. + tokio::spawn(async move { while rx.recv().await.is_some() {} }); + registry_for_register.register( + "sbx".to_string(), + "s1".to_string(), + tx, + make_shutdown(), + ); + }); + + let result = registry.open_relay("sbx", Duration::from_secs(2)).await; + assert!( + result.is_ok(), + "open_relay should succeed when session arrives mid-wait: {result:?}" + ); + } + + #[tokio::test] + async fn open_relay_fails_when_session_receiver_dropped() { + let registry = SupervisorSessionRegistry::new(); + let (tx, rx) = mpsc::channel::(4); + registry.register("sbx".to_string(), "s1".to_string(), tx, make_shutdown()); + + // Simulate the supervisor's stream going away between lookup and send: + // the receiver held by `ReceiverStream` is dropped. + drop(rx); + + let err = registry + .open_relay("sbx", Duration::from_secs(1)) + .await + .expect_err("open_relay should fail when mpsc is closed"); + assert_eq!(err.code(), tonic::Code::Unavailable); + // The pending-relay entry must have been cleaned up on failure. + assert!(registry.pending_relays.lock().unwrap().is_empty()); + } + + #[tokio::test] + async fn open_relay_rejects_when_global_cap_reached() { + let registry = SupervisorSessionRegistry::new(); + let (tx, _rx) = mpsc::channel::(8); + registry.register( + "sbx-a".to_string(), + "s-a".to_string(), + tx.clone(), + make_shutdown(), + ); + registry.register("sbx-b".to_string(), "s-b".to_string(), tx, make_shutdown()); + + // Pre-seed pending_relays to exactly the global cap, split across two + // sandboxes so neither hits the per-sandbox cap first. + { + let mut pending = registry.pending_relays.lock().unwrap(); + for i in 0..MAX_PENDING_RELAYS { + let (oneshot_tx, _) = oneshot::channel(); + let sandbox_id = if i % 2 == 0 { "sbx-a" } else { "sbx-b" }; + pending.insert( + format!("channel-{i}"), + PendingRelay { + sender: oneshot_tx, + sandbox_id: sandbox_id.to_string(), + created_at: Instant::now(), + }, + ); + } + } + + let err = registry + .open_relay("sbx-a", Duration::from_millis(50)) + .await + .expect_err("open_relay should reject once global cap is reached"); + assert_eq!(err.code(), tonic::Code::ResourceExhausted); + assert!(err.message().contains("gateway relay capacity")); + } + + #[tokio::test] + async fn open_relay_rejects_when_per_sandbox_cap_reached() { + let registry = SupervisorSessionRegistry::new(); + let (tx, _rx) = mpsc::channel::(8); + registry.register("sbx".to_string(), "s".to_string(), tx, make_shutdown()); + + { + let mut pending = registry.pending_relays.lock().unwrap(); + for i in 0..MAX_PENDING_RELAYS_PER_SANDBOX { + let (oneshot_tx, _) = oneshot::channel(); + pending.insert( + format!("channel-{i}"), + PendingRelay { + sender: oneshot_tx, + sandbox_id: "sbx".to_string(), + created_at: Instant::now(), + }, + ); + } + } + + let err = registry + .open_relay("sbx", Duration::from_millis(50)) + .await + .expect_err("open_relay should reject when per-sandbox cap is reached"); + assert_eq!(err.code(), tonic::Code::ResourceExhausted); + assert!(err.message().contains("per-sandbox relay limit")); + + // A different sandbox still has headroom. + let (tx2, _rx2) = mpsc::channel::(8); + registry.register( + "sbx-other".to_string(), + "s-other".to_string(), + tx2, + make_shutdown(), + ); + registry + .open_relay("sbx-other", Duration::from_millis(50)) + .await + .expect("different sandbox should still accept new relays"); + } + + #[tokio::test] + async fn open_relay_uses_newest_session_after_supersede() { + use tokio::sync::mpsc::error::TryRecvError; + + let registry = SupervisorSessionRegistry::new(); + let (tx_old, mut rx_old) = mpsc::channel::(4); + let (tx_new, mut rx_new) = mpsc::channel(4); + + // Hold a clone of the old sender so supersede doesn't close the old + // channel — that way try_recv distinguishes "no message sent" from + // "channel closed". + let _tx_old_alive = tx_old.clone(); + + registry.register( + "sbx".to_string(), + "s-old".to_string(), + tx_old, + make_shutdown(), + ); + registry.register( + "sbx".to_string(), + "s-new".to_string(), + tx_new, + make_shutdown(), + ); + + let (_channel_id, _relay_rx) = registry + .open_relay("sbx", Duration::from_secs(1)) + .await + .expect("open_relay should succeed"); + + let msg = rx_new + .recv() + .await + .expect("new session should receive RelayOpen"); + assert!(matches!( + msg.payload, + Some(gateway_message::Payload::RelayOpen(_)) + )); + + // The old session must have received no messages — the channel is + // still open but empty. + match rx_old.try_recv() { + Err(TryRecvError::Empty) => {} + other => panic!("expected Empty on superseded session, got {other:?}"), + } + } + + #[tokio::test] + async fn register_signals_shutdown_to_previous_session() { + let registry = SupervisorSessionRegistry::new(); + let (tx_old, _rx_old) = mpsc::channel::(1); + let (tx_new, _rx_new) = mpsc::channel::(1); + + let (shutdown_tx, shutdown_rx) = oneshot::channel::<()>(); + registry.register("sbx".to_string(), "s-old".to_string(), tx_old, shutdown_tx); + + // Supersede with a new session — register must fire the old session's + // shutdown signal so its task can exit and drop its tx clone. + let superseded = registry.register( + "sbx".to_string(), + "s-new".to_string(), + tx_new, + make_shutdown(), + ); + assert!(superseded, "second register should report supersede"); + + // The old session's shutdown receiver must now resolve. + shutdown_rx + .await + .expect("shutdown signal should arrive at superseded session"); + } + + #[tokio::test] + async fn replay_pending_relays_reissues_open_to_superseding_session() { + let registry = SupervisorSessionRegistry::new(); + let (tx_old, mut rx_old) = mpsc::channel::(4); + let (tx_new, mut rx_new) = mpsc::channel::(4); + + registry.register( + "sbx".to_string(), + "s-old".to_string(), + tx_old, + make_shutdown(), + ); + + let (channel_id, _relay_rx) = registry + .open_relay("sbx", Duration::from_secs(1)) + .await + .expect("open_relay should succeed"); + + let original = rx_old + .recv() + .await + .expect("old session should receive initial RelayOpen"); + assert!(matches!( + original.payload, + Some(gateway_message::Payload::RelayOpen(_)) + )); + + let superseded = registry.register( + "sbx".to_string(), + "s-new".to_string(), + tx_new, + make_shutdown(), + ); + assert!(superseded); + + registry + .replay_pending_relays("sbx", ®istry.lookup_session("sbx").unwrap()) + .await; + + let replayed = rx_new + .recv() + .await + .expect("new session should receive replayed RelayOpen"); + match replayed.payload { + Some(gateway_message::Payload::RelayOpen(open)) => { + assert_eq!(open.channel_id, channel_id); + } + other => panic!("expected RelayOpen on replay, got {other:?}"), + } + } + + #[tokio::test] + async fn require_persisted_sandbox_rejects_missing_sandbox() { + let store = Arc::new( + Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(), + ); + + let err = require_persisted_sandbox(&store, "missing") + .await + .expect_err("missing sandbox should be rejected"); + + assert_eq!(err.code(), tonic::Code::NotFound); + } + + #[tokio::test] + async fn require_persisted_sandbox_accepts_existing_sandbox() { + let store = Arc::new( + Store::connect("sqlite::memory:?cache=shared") + .await + .unwrap(), + ); + store + .put_message(&sandbox_record("sbx-1", "sandbox-one")) + .await + .expect("sandbox should persist"); + + require_persisted_sandbox(&store, "sbx-1") + .await + .expect("persisted sandbox should be accepted"); + } + + // ---- claim_relay: expiry, drop, wiring ---- + + #[test] + fn claim_relay_unknown_channel() { + let registry = SupervisorSessionRegistry::new(); + let err = registry.claim_relay("nonexistent").expect_err("should err"); + assert_eq!(err.code(), tonic::Code::NotFound); + } + + #[test] + fn claim_relay_success() { + let registry = SupervisorSessionRegistry::new(); + let (relay_tx, _relay_rx) = oneshot::channel(); + registry.pending_relays.lock().unwrap().insert( + "ch-1".to_string(), + PendingRelay { + sender: relay_tx, + sandbox_id: "sbx-test".to_string(), + created_at: Instant::now(), + }, + ); + + let result = registry.claim_relay("ch-1"); + assert!(result.is_ok()); + assert!(!registry.pending_relays.lock().unwrap().contains_key("ch-1")); + } + + #[test] + fn claim_relay_expired_returns_deadline_exceeded() { + let registry = SupervisorSessionRegistry::new(); + let (relay_tx, _relay_rx) = oneshot::channel(); + registry.pending_relays.lock().unwrap().insert( + "ch-old".to_string(), + PendingRelay { + sender: relay_tx, + sandbox_id: "sbx-test".to_string(), + created_at: Instant::now() + .checked_sub(Duration::from_secs(60)) + .expect("test instant subtraction underflow"), + }, + ); + + let err = registry + .claim_relay("ch-old") + .expect_err("expired entry must fail"); + assert_eq!(err.code(), tonic::Code::DeadlineExceeded); + // Entry must have been consumed regardless. + assert!( + !registry + .pending_relays + .lock() + .unwrap() + .contains_key("ch-old") + ); + } + + #[test] + fn claim_relay_receiver_dropped_returns_internal() { + let registry = SupervisorSessionRegistry::new(); + let (relay_tx, relay_rx) = oneshot::channel::(); + drop(relay_rx); // Gateway-side waiter has given up already. + registry.pending_relays.lock().unwrap().insert( + "ch-1".to_string(), + PendingRelay { + sender: relay_tx, + sandbox_id: "sbx-test".to_string(), + created_at: Instant::now(), + }, + ); + + let err = registry + .claim_relay("ch-1") + .expect_err("should err when receiver is gone"); + assert_eq!(err.code(), tonic::Code::Internal); + } + + #[tokio::test] + async fn claim_relay_connects_both_ends() { + let registry = SupervisorSessionRegistry::new(); + let (relay_tx, relay_rx) = oneshot::channel::(); + registry.pending_relays.lock().unwrap().insert( + "ch-io".to_string(), + PendingRelay { + sender: relay_tx, + sandbox_id: "sbx-test".to_string(), + created_at: Instant::now(), + }, + ); + + let mut supervisor_side = registry.claim_relay("ch-io").expect("claim should succeed"); + let mut gateway_side = relay_rx.await.expect("gateway side should receive stream"); + + // Supervisor side writes → gateway side reads. + supervisor_side.write_all(b"hello").await.unwrap(); + let mut buf = [0u8; 5]; + gateway_side.read_exact(&mut buf).await.unwrap(); + assert_eq!(&buf, b"hello"); + + // Gateway side writes → supervisor side reads. + gateway_side.write_all(b"world").await.unwrap(); + let mut buf = [0u8; 5]; + supervisor_side.read_exact(&mut buf).await.unwrap(); + assert_eq!(&buf, b"world"); + } + + // ---- reap_expired_relays ---- + + #[test] + fn reap_expired_relays_removes_old_entries() { + let registry = SupervisorSessionRegistry::new(); + let (relay_tx, _relay_rx) = oneshot::channel(); + registry.pending_relays.lock().unwrap().insert( + "ch-old".to_string(), + PendingRelay { + sender: relay_tx, + sandbox_id: "sbx-test".to_string(), + created_at: Instant::now() + .checked_sub(Duration::from_secs(60)) + .expect("test instant subtraction underflow"), + }, + ); + + registry.reap_expired_relays(); + assert!( + !registry + .pending_relays + .lock() + .unwrap() + .contains_key("ch-old") + ); + } + + #[test] + fn reap_expired_relays_keeps_fresh_entries() { + let registry = SupervisorSessionRegistry::new(); + let (relay_tx, _relay_rx) = oneshot::channel(); + registry.pending_relays.lock().unwrap().insert( + "ch-fresh".to_string(), + PendingRelay { + sender: relay_tx, + sandbox_id: "sbx-test".to_string(), + created_at: Instant::now(), + }, + ); + + registry.reap_expired_relays(); + assert!( + registry + .pending_relays + .lock() + .unwrap() + .contains_key("ch-fresh") + ); + } +} diff --git a/crates/openshell-server/src/tracing_bus.rs b/crates/openshell-server/src/tracing_bus.rs index 6767a450ee..cf168e3063 100644 --- a/crates/openshell-server/src/tracing_bus.rs +++ b/crates/openshell-server/src/tracing_bus.rs @@ -8,6 +8,7 @@ use std::sync::{Arc, Mutex}; use std::time::{SystemTime, UNIX_EPOCH}; use openshell_core::proto::{SandboxLogLine, SandboxStreamEvent}; +use openshell_ocsf::OCSF_TARGET; use tokio::sync::broadcast; use tracing::{Event, Subscriber}; use tracing_subscriber::layer::Context; @@ -147,12 +148,13 @@ where }; let msg = visitor.message.unwrap_or_else(|| meta.name().to_string()); + let level = display_level(meta.target(), &meta.level().to_string()); let ts = current_time_ms().unwrap_or(0); let log = SandboxLogLine { sandbox_id: sandbox_id.clone(), timestamp_ms: ts, - level: meta.level().to_string(), + level, target: meta.target().to_string(), message: msg, source: "gateway".to_string(), @@ -196,6 +198,14 @@ fn current_time_ms() -> Option { i64::try_from(now.as_millis()).ok() } +fn display_level(target: &str, level: &str) -> String { + if target == OCSF_TARGET { + "OCSF".to_string() + } else { + level.to_string() + } +} + #[cfg(test)] mod tests { use super::*; @@ -274,6 +284,12 @@ mod tests { bus.remove("nonexistent"); } + #[test] + fn display_level_maps_ocsf_target_to_ocsf() { + assert_eq!(display_level(OCSF_TARGET, "INFO"), "OCSF"); + assert_eq!(display_level("openshell_server", "WARN"), "WARN"); + } + #[test] fn platform_event_bus_remove_cleans_up() { let bus = PlatformEventBus::new(); diff --git a/crates/openshell-server/tests/auth_endpoint_integration.rs b/crates/openshell-server/tests/auth_endpoint_integration.rs index 7c6545873c..e5f9dc4e97 100644 --- a/crates/openshell-server/tests/auth_endpoint_integration.rs +++ b/crates/openshell-server/tests/auth_endpoint_integration.rs @@ -508,6 +508,22 @@ impl openshell_core::proto::open_shell_server::OpenShell for TestOpenShell { Err(tonic::Status::unimplemented("test")) } + async fn list_provider_profiles( + &self, + _: tonic::Request, + ) -> Result, tonic::Status> + { + Err(tonic::Status::unimplemented("test")) + } + + async fn get_provider_profile( + &self, + _: tonic::Request, + ) -> Result, tonic::Status> + { + Err(tonic::Status::unimplemented("test")) + } + async fn update_provider( &self, _: tonic::Request, @@ -528,6 +544,9 @@ impl openshell_core::proto::open_shell_server::OpenShell for TestOpenShell { type ExecSandboxStream = tokio_stream::wrappers::ReceiverStream< Result, >; + type ConnectSupervisorStream = tokio_stream::wrappers::ReceiverStream< + Result, + >; async fn watch_sandbox( &self, @@ -663,6 +682,24 @@ impl openshell_core::proto::open_shell_server::OpenShell for TestOpenShell { { Err(tonic::Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, tonic::Status> { + Err(tonic::Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = tokio_stream::wrappers::ReceiverStream< + Result, + >; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, tonic::Status> { + Err(tonic::Status::unimplemented("not implemented in test")) + } } /// Test 7: Plaintext server (no TLS) accepts both gRPC and HTTP. diff --git a/crates/openshell-server/tests/edge_tunnel_auth.rs b/crates/openshell-server/tests/edge_tunnel_auth.rs index 22f08434d6..ed6ed398f5 100644 --- a/crates/openshell-server/tests/edge_tunnel_auth.rs +++ b/crates/openshell-server/tests/edge_tunnel_auth.rs @@ -12,7 +12,7 @@ //! //! Test matrix: //! -//! | allow_unauthenticated | client cert | bearer auth header | expected | +//! | `allow_unauthenticated` | client cert | bearer auth header | expected | //! |-----------------------|-------------|--------------------|----------| //! | false | valid | — | OK | //! | false | none | — | rejected | @@ -37,13 +37,14 @@ use hyper_util::{ use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - ExecSandboxEvent, ExecSandboxRequest, GetGatewayConfigRequest, GetGatewayConfigResponse, - GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, - GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxesRequest, ListSandboxesResponse, ProviderResponse, RevokeSshSessionRequest, - RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, ServiceStatus, - UpdateProviderRequest, WatchSandboxRequest, + ExecSandboxEvent, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, + GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, + ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, + SandboxStreamEvent, ServiceStatus, SupervisorMessage, UpdateProviderRequest, + WatchSandboxRequest, open_shell_client::OpenShellClient, open_shell_server::{OpenShell, OpenShellServer}, }; @@ -68,7 +69,7 @@ fn install_rustls_provider() { let _ = rustls::crypto::ring::default_provider().install_default(); } -/// Minimal OpenShell implementation for testing. +/// Minimal `OpenShell` implementation for testing. #[derive(Clone, Default)] struct TestOpenShell; @@ -170,6 +171,20 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("not implemented in test")) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, _request: tonic::Request, @@ -186,6 +201,7 @@ impl OpenShell for TestOpenShell { type WatchSandboxStream = ReceiverStream>; type ExecSandboxStream = ReceiverStream>; + type ConnectSupervisorStream = ReceiverStream>; async fn watch_sandbox( &self, @@ -307,6 +323,22 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } // --------------------------------------------------------------------------- @@ -555,7 +587,7 @@ fn https_client_no_cert( // Tests // =========================================================================== -/// Baseline: with allow_unauthenticated=false (default), mTLS connections work. +/// Baseline: with `allow_unauthenticated=false` (default), mTLS connections work. #[tokio::test] async fn baseline_mtls_works_with_mandatory_client_certs() { install_rustls_provider(); @@ -595,7 +627,7 @@ async fn baseline_mtls_works_with_mandatory_client_certs() { server.abort(); } -/// Baseline: with allow_unauthenticated=false, no-client-cert connections are +/// Baseline: with `allow_unauthenticated=false`, no-client-cert connections are /// rejected at the TLS layer. #[tokio::test] async fn baseline_no_cert_rejected_with_mandatory_mtls() { @@ -635,7 +667,7 @@ async fn baseline_no_cert_rejected_with_mandatory_mtls() { server.abort(); } -/// With allow_unauthenticated=true, mTLS connections still work (dual-auth). +/// With `allow_unauthenticated=true`, mTLS connections still work (dual-auth). #[tokio::test] async fn dual_auth_mtls_still_accepted() { install_rustls_provider(); @@ -675,7 +707,7 @@ async fn dual_auth_mtls_still_accepted() { server.abort(); } -/// With allow_unauthenticated=true, no-client-cert connections pass the TLS +/// With `allow_unauthenticated=true`, no-client-cert connections pass the TLS /// handshake. This simulates Cloudflare Tunnel re-originating a connection. /// /// The gRPC health check succeeds because there is no auth middleware yet — @@ -757,7 +789,7 @@ async fn tunnel_mode_cf_authorization_header_reaches_server() { server.abort(); } -/// With allow_unauthenticated=true, a client cert from a rogue CA is still +/// With `allow_unauthenticated=true`, a client cert from a rogue CA is still /// rejected by the TLS layer — the verifier still validates presented certs. #[tokio::test] async fn tunnel_mode_rogue_cert_still_rejected() { diff --git a/crates/openshell-server/tests/multiplex_integration.rs b/crates/openshell-server/tests/multiplex_integration.rs index 1957c5b87d..c91dd60615 100644 --- a/crates/openshell-server/tests/multiplex_integration.rs +++ b/crates/openshell-server/tests/multiplex_integration.rs @@ -11,13 +11,14 @@ use hyper_util::{ use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - ExecSandboxEvent, ExecSandboxRequest, GetGatewayConfigRequest, GetGatewayConfigResponse, - GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, - GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxesRequest, ListSandboxesResponse, ProviderResponse, RevokeSshSessionRequest, - RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, ServiceStatus, - UpdateProviderRequest, WatchSandboxRequest, + ExecSandboxEvent, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, + GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, + ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, + SandboxStreamEvent, ServiceStatus, SupervisorMessage, UpdateProviderRequest, + WatchSandboxRequest, open_shell_client::OpenShellClient, open_shell_server::{OpenShell, OpenShellServer}, }; @@ -134,6 +135,20 @@ impl OpenShell for TestOpenShell { )) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, _request: tonic::Request, @@ -154,6 +169,7 @@ impl OpenShell for TestOpenShell { type WatchSandboxStream = ReceiverStream>; type ExecSandboxStream = ReceiverStream>; + type ConnectSupervisorStream = ReceiverStream>; async fn watch_sandbox( &self, @@ -275,6 +291,22 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } #[tokio::test] @@ -325,3 +357,76 @@ async fn serves_grpc_and_http_on_same_port() { server.abort(); } + +/// Verify tonic metadata ↔ HTTP header roundtrip for `x-request-id`. +/// +/// This intentionally constructs its own request-ID layers from +/// `tower-http`'s public API rather than reusing the production macro +/// (which is crate-private). Production middleware composition and +/// layer ordering are covered by the unit tests in `multiplex::tests`. +#[tokio::test] +async fn grpc_response_propagates_request_id() { + use tower::ServiceBuilder; + use tower_http::request_id::{ + MakeRequestId, PropagateRequestIdLayer, RequestId, SetRequestIdLayer, + }; + + #[derive(Clone)] + struct TestUuidRequestId; + + impl MakeRequestId for TestUuidRequestId { + fn make_request_id(&mut self, _req: &Request) -> Option { + let id = uuid::Uuid::new_v4().to_string(); + Some(RequestId::new(http::HeaderValue::from_str(&id).unwrap())) + } + } + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + + let x_request_id = http::HeaderName::from_static("x-request-id"); + let grpc_service = ServiceBuilder::new() + .layer(SetRequestIdLayer::new( + x_request_id.clone(), + TestUuidRequestId, + )) + .layer(PropagateRequestIdLayer::new(x_request_id)) + .service(OpenShellServer::new(TestOpenShell)); + let http_service = health_router(); + let service = MultiplexedService::new(grpc_service, http_service); + + tokio::spawn(async move { + loop { + let Ok((stream, _)) = listener.accept().await else { + continue; + }; + let svc = service.clone(); + tokio::spawn(async move { + let _ = Builder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(stream), svc) + .await; + }); + } + }); + + let mut client = OpenShellClient::connect(format!("http://{addr}")) + .await + .unwrap(); + + // Server generates a UUID when client omits x-request-id. + let response = client.health(HealthRequest {}).await.unwrap(); + let generated = response + .metadata() + .get("x-request-id") + .expect("gRPC response should include server-generated x-request-id"); + uuid::Uuid::parse_str(generated.to_str().unwrap()).expect("should be a valid UUID"); + + // Server preserves a client-supplied x-request-id. + let mut request = tonic::Request::new(HealthRequest {}); + request + .metadata_mut() + .insert("x-request-id", "grpc-corr-id".parse().unwrap()); + let response = client.health(request).await.unwrap(); + let echoed = response.metadata().get("x-request-id").unwrap(); + assert_eq!(echoed.to_str().unwrap(), "grpc-corr-id"); +} diff --git a/crates/openshell-server/tests/multiplex_tls_integration.rs b/crates/openshell-server/tests/multiplex_tls_integration.rs index 98d5d62563..6942d66f74 100644 --- a/crates/openshell-server/tests/multiplex_tls_integration.rs +++ b/crates/openshell-server/tests/multiplex_tls_integration.rs @@ -13,13 +13,14 @@ use hyper_util::{ use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - ExecSandboxEvent, ExecSandboxRequest, GetGatewayConfigRequest, GetGatewayConfigResponse, - GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, - GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxesRequest, ListSandboxesResponse, ProviderResponse, RevokeSshSessionRequest, - RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, ServiceStatus, - UpdateProviderRequest, WatchSandboxRequest, + ExecSandboxEvent, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, + GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, + ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, + SandboxStreamEvent, ServiceStatus, SupervisorMessage, UpdateProviderRequest, + WatchSandboxRequest, open_shell_client::OpenShellClient, open_shell_server::{OpenShell, OpenShellServer}, }; @@ -147,6 +148,20 @@ impl OpenShell for TestOpenShell { )) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, _request: tonic::Request, @@ -167,6 +182,7 @@ impl OpenShell for TestOpenShell { type WatchSandboxStream = ReceiverStream>; type ExecSandboxStream = ReceiverStream>; + type ConnectSupervisorStream = ReceiverStream>; async fn watch_sandbox( &self, @@ -288,6 +304,22 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } /// PKI bundle: CA cert, server cert+key, client cert+key. diff --git a/crates/openshell-server/tests/supervisor_relay_integration.rs b/crates/openshell-server/tests/supervisor_relay_integration.rs new file mode 100644 index 0000000000..d77cfd375b --- /dev/null +++ b/crates/openshell-server/tests/supervisor_relay_integration.rs @@ -0,0 +1,587 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Integration tests for the supervisor relay gRPC path. +//! +//! Stands up an in-process tonic server hosting the real `handle_relay_stream` +//! handler, plus a mock "supervisor" client that calls `relay_stream` over a +//! real `Channel`. Exercises the wire contract (typed `RelayFrame { Init | Data }`), +//! `SupervisorSessionRegistry::open_relay` → `claim_relay` pairing, and the +//! bidirectional byte bridge inside the handler. +//! +//! These tests complement the unit tests in `supervisor_session.rs` (which +//! exercise registry state only) and the live cluster tests (which exercise +//! the full CLI → gateway → sandbox path). They catch regressions in the gRPC +//! wire layer that unit tests can't see and that are expensive to catch in +//! E2E. + +use std::sync::Arc; +use std::time::Duration; + +use hyper_util::{ + rt::{TokioExecutor, TokioIo}, + server::conn::auto::Builder, +}; +use openshell_core::proto::{ + GatewayMessage, RelayFrame, RelayInit, SupervisorMessage, + open_shell_client::OpenShellClient, + open_shell_server::{OpenShell, OpenShellServer}, +}; +use openshell_server::supervisor_session::SupervisorSessionRegistry; +use openshell_server::{MultiplexedService, health_router}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpListener; +use tokio::sync::mpsc; +use tokio_stream::StreamExt; +use tokio_stream::wrappers::ReceiverStream; +use tonic::transport::{Channel, Endpoint}; +use tonic::{Response, Status}; + +// --------------------------------------------------------------------------- +// Gateway service: only relay_stream does real work; everything else stubs. +// --------------------------------------------------------------------------- + +#[derive(Clone)] +struct RelayGateway { + registry: Arc, +} + +#[tonic::async_trait] +impl OpenShell for RelayGateway { + type RelayStreamStream = std::pin::Pin< + Box> + Send + 'static>, + >; + + async fn relay_stream( + &self, + request: tonic::Request>, + ) -> Result, Status> { + openshell_server::supervisor_session::handle_relay_stream(&self.registry, request).await + } + + // ------ unused stubs ------ + + type ConnectSupervisorStream = ReceiverStream>; + async fn connect_supervisor( + &self, + _: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + type WatchSandboxStream = + ReceiverStream>; + async fn watch_sandbox( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + type ExecSandboxStream = + ReceiverStream>; + async fn exec_sandbox( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn health( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn create_sandbox( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_sandbox( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn list_sandboxes( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn delete_sandbox( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_sandbox_config( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_gateway_config( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_sandbox_provider_environment( + &self, + _: tonic::Request, + ) -> Result, Status> + { + Err(Status::unimplemented("unused")) + } + async fn create_ssh_session( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn revoke_ssh_session( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn create_provider( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn update_provider( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_provider( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn list_providers( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn list_provider_profiles( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn get_provider_profile( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn delete_provider( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn update_config( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_sandbox_policy_status( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn list_sandbox_policies( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn report_policy_status( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_sandbox_logs( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn push_sandbox_logs( + &self, + _: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn submit_policy_analysis( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_draft_policy( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn approve_draft_chunk( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn reject_draft_chunk( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn approve_all_draft_chunks( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn edit_draft_chunk( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn undo_draft_chunk( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn clear_draft_chunks( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn get_draft_history( + &self, + _: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } +} + +// --------------------------------------------------------------------------- +// Test harness +// --------------------------------------------------------------------------- + +async fn spawn_gateway(registry: Arc) -> Channel { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + + let grpc = OpenShellServer::new(RelayGateway { registry }); + let service = MultiplexedService::new(grpc, health_router()); + + tokio::spawn(async move { + loop { + let Ok((stream, _)) = listener.accept().await else { + continue; + }; + let svc = service.clone(); + tokio::spawn(async move { + let _ = Builder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(stream), svc) + .await; + }); + } + }); + + Endpoint::from_shared(format!("http://{addr}")) + .unwrap() + .connect() + .await + .expect("client connect") +} + +fn register_session( + registry: &SupervisorSessionRegistry, + sandbox_id: &str, +) -> mpsc::Receiver { + register_session_with_capacity(registry, sandbox_id, 8) +} + +fn register_session_with_capacity( + registry: &SupervisorSessionRegistry, + sandbox_id: &str, + capacity: usize, +) -> mpsc::Receiver { + let (tx, rx) = mpsc::channel(capacity); + let (shutdown_tx, _shutdown_rx) = tokio::sync::oneshot::channel::<()>(); + registry.register( + sandbox_id.to_string(), + "sess-1".to_string(), + tx, + shutdown_tx, + ); + rx +} + +/// Mock supervisor that opens a `RelayStream`, sends `Init`, then echoes every +/// data frame it receives. Returns when the gateway drops the stream or when +/// the supervisor's own outbound channel closes. +async fn run_echo_supervisor(channel: Channel, channel_id: String) { + let mut client = OpenShellClient::new(channel); + let (out_tx, out_rx) = mpsc::channel::(16); + let outbound = ReceiverStream::new(out_rx); + + out_tx + .send(RelayFrame { + payload: Some(openshell_core::proto::relay_frame::Payload::Init( + RelayInit { channel_id }, + )), + }) + .await + .expect("send init"); + + let response = client + .relay_stream(outbound) + .await + .expect("relay_stream rpc"); + let mut inbound = response.into_inner(); + + while let Some(msg) = inbound.next().await { + let Ok(frame) = msg else { break }; + let Some(openshell_core::proto::relay_frame::Payload::Data(data)) = frame.payload else { + continue; + }; + let echoed = RelayFrame { + payload: Some(openshell_core::proto::relay_frame::Payload::Data(data)), + }; + if out_tx.send(echoed).await.is_err() { + break; + } + } +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn relay_round_trips_bytes() { + let registry = Arc::new(SupervisorSessionRegistry::new()); + let channel = spawn_gateway(Arc::clone(®istry)).await; + let mut session_rx = register_session(®istry, "sbx"); + + let (channel_id, relay_rx) = registry + .open_relay("sbx", Duration::from_secs(2)) + .await + .expect("open_relay"); + + let opened = match session_rx.recv().await.expect("RelayOpen").payload { + Some(openshell_core::proto::gateway_message::Payload::RelayOpen(r)) => r.channel_id, + other => panic!("expected RelayOpen, got {other:?}"), + }; + assert_eq!(opened, channel_id); + + tokio::spawn(run_echo_supervisor(channel, channel_id)); + + let relay = relay_rx.await.expect("relay duplex"); + let (mut read_half, mut write_half) = tokio::io::split(relay); + + write_half.write_all(b"hello relay").await.expect("write"); + write_half.flush().await.expect("flush"); + + let mut buf = [0u8; 11]; + read_half.read_exact(&mut buf).await.expect("read echoed"); + assert_eq!(&buf, b"hello relay"); +} + +#[tokio::test] +async fn relay_closes_cleanly_when_gateway_drops() { + let registry = Arc::new(SupervisorSessionRegistry::new()); + let channel = spawn_gateway(Arc::clone(®istry)).await; + let mut session_rx = register_session(®istry, "sbx"); + + let (channel_id, relay_rx) = registry + .open_relay("sbx", Duration::from_secs(2)) + .await + .expect("open_relay"); + let _ = session_rx.recv().await.expect("RelayOpen"); + + let supervisor = tokio::spawn(run_echo_supervisor(channel, channel_id)); + + let relay = relay_rx.await.expect("relay duplex"); + drop(relay); + + // The supervisor's inbound stream should terminate shortly after the + // gateway side drops — its echo loop exits and the task finishes. + tokio::time::timeout(Duration::from_secs(5), supervisor) + .await + .expect("supervisor should terminate after gateway drop") + .expect("supervisor task"); +} + +#[tokio::test] +async fn relay_sees_eof_when_supervisor_closes() { + let registry = Arc::new(SupervisorSessionRegistry::new()); + let channel = spawn_gateway(Arc::clone(®istry)).await; + let mut session_rx = register_session(®istry, "sbx"); + + let (channel_id, relay_rx) = registry + .open_relay("sbx", Duration::from_secs(2)) + .await + .expect("open_relay"); + let _ = session_rx.recv().await.expect("RelayOpen"); + + // Supervisor sends init, then drops its outbound sender → gateway reader + // should see EOF. + let supervisor = { + let channel_id = channel_id.clone(); + tokio::spawn(async move { + let mut client = OpenShellClient::new(channel); + let (out_tx, out_rx) = mpsc::channel::(4); + let outbound = ReceiverStream::new(out_rx); + out_tx + .send(RelayFrame { + payload: Some(openshell_core::proto::relay_frame::Payload::Init( + RelayInit { channel_id }, + )), + }) + .await + .expect("send init"); + let _response = client.relay_stream(outbound).await.expect("rpc"); + drop(out_tx); + tokio::time::sleep(Duration::from_millis(200)).await; + }) + }; + + let relay = relay_rx.await.expect("relay duplex"); + let (mut read_half, _write_half) = tokio::io::split(relay); + let mut buf = [0u8; 16]; + let n = tokio::time::timeout(Duration::from_secs(5), read_half.read(&mut buf)) + .await + .expect("read should complete") + .expect("read ok"); + assert_eq!(n, 0, "gateway-side read should see EOF"); + + supervisor.await.expect("supervisor task"); +} + +#[tokio::test] +async fn open_relay_times_out_when_no_session() { + let registry = Arc::new(SupervisorSessionRegistry::new()); + let _channel = spawn_gateway(Arc::clone(®istry)).await; + + let err = registry + .open_relay("missing", Duration::from_millis(100)) + .await + .expect_err("should time out"); + assert_eq!(err.code(), tonic::Code::Unavailable); +} + +#[tokio::test] +async fn concurrent_relays_multiplex_independently() { + let registry = Arc::new(SupervisorSessionRegistry::new()); + let channel = spawn_gateway(Arc::clone(®istry)).await; + let mut session_rx = register_session(®istry, "sbx"); + + let (id_a, rx_a) = registry + .open_relay("sbx", Duration::from_secs(2)) + .await + .expect("open_relay a"); + let _ = session_rx.recv().await.expect("RelayOpen a"); + + let (id_b, rx_b) = registry + .open_relay("sbx", Duration::from_secs(2)) + .await + .expect("open_relay b"); + let _ = session_rx.recv().await.expect("RelayOpen b"); + assert_ne!(id_a, id_b); + + tokio::spawn(run_echo_supervisor(channel.clone(), id_a)); + tokio::spawn(run_echo_supervisor(channel, id_b)); + + let relay_a = rx_a.await.expect("relay a"); + let relay_b = rx_b.await.expect("relay b"); + + let (mut ra, mut wa) = tokio::io::split(relay_a); + let (mut rb, mut wb) = tokio::io::split(relay_b); + + wa.write_all(b"stream-A").await.unwrap(); + wb.write_all(b"stream-B").await.unwrap(); + wa.flush().await.unwrap(); + wb.flush().await.unwrap(); + + let mut buf_a = [0u8; 8]; + let mut buf_b = [0u8; 8]; + ra.read_exact(&mut buf_a).await.unwrap(); + rb.read_exact(&mut buf_b).await.unwrap(); + assert_eq!(&buf_a, b"stream-A"); + assert_eq!(&buf_b, b"stream-B"); +} + +/// Bursts more `open_relay` calls than the per-sandbox cap allows in parallel +/// and asserts the registry enforces the ceiling cleanly. A well-behaved +/// caller inside the cap still succeeds; overflow calls return `ResourceExhausted` +/// rather than racing the pending map into an inconsistent state. +#[tokio::test] +async fn open_relay_enforces_per_sandbox_cap_under_concurrent_burst() { + let registry = Arc::new(SupervisorSessionRegistry::new()); + let _channel = spawn_gateway(Arc::clone(®istry)).await; + // Oversized mpsc so the session doesn't backpressure the burst — the cap, + // not the channel, is what we're testing. + let _session_rx = register_session_with_capacity(®istry, "sbx", 256); + + // Fire 64 concurrent opens. Per-sandbox cap is 32, global cap is 256, + // so exactly 32 should succeed and 32 should be rejected with + // `ResourceExhausted` carrying the per-sandbox message. + let mut handles = Vec::with_capacity(64); + for _ in 0..64 { + let r = Arc::clone(®istry); + handles.push(tokio::spawn(async move { + r.open_relay("sbx", Duration::from_secs(1)).await + })); + } + + let mut ok = 0usize; + let mut exhausted = 0usize; + for h in handles { + match h.await.expect("task joined") { + Ok(_pair) => ok += 1, + Err(status) if status.code() == tonic::Code::ResourceExhausted => { + assert!( + status.message().contains("per-sandbox relay limit"), + "expected per-sandbox error message, got: {}", + status.message() + ); + exhausted += 1; + } + Err(other) => panic!("unexpected open_relay error: {other:?}"), + } + } + assert_eq!(ok, 32, "exactly per-sandbox cap should succeed"); + assert_eq!(exhausted, 32, "overflow should be rejected, not dropped"); + + // A different sandbox still has headroom — the per-sandbox cap doesn't + // leak onto unrelated tenants. + let _other_rx = register_session_with_capacity(®istry, "sbx-other", 8); + registry + .open_relay("sbx-other", Duration::from_secs(1)) + .await + .expect("other sandbox should not be affected by sbx cap"); +} diff --git a/crates/openshell-server/tests/ws_tunnel_integration.rs b/crates/openshell-server/tests/ws_tunnel_integration.rs index 54a7354c86..f196edb07e 100644 --- a/crates/openshell-server/tests/ws_tunnel_integration.rs +++ b/crates/openshell-server/tests/ws_tunnel_integration.rs @@ -40,13 +40,14 @@ use hyper_util::{ use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - ExecSandboxEvent, ExecSandboxRequest, GetGatewayConfigRequest, GetGatewayConfigResponse, - GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, - GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxesRequest, ListSandboxesResponse, ProviderResponse, RevokeSshSessionRequest, - RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, ServiceStatus, - UpdateProviderRequest, WatchSandboxRequest, + ExecSandboxEvent, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, + GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, + GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, + ListProvidersRequest, ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, + ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, + SandboxStreamEvent, ServiceStatus, SupervisorMessage, UpdateProviderRequest, + WatchSandboxRequest, open_shell_client::OpenShellClient, open_shell_server::{OpenShell, OpenShellServer}, }; @@ -164,6 +165,20 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("not implemented in test")) } + async fn list_provider_profiles( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + async fn get_provider_profile( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + async fn update_provider( &self, _request: tonic::Request, @@ -180,6 +195,7 @@ impl OpenShell for TestOpenShell { type WatchSandboxStream = ReceiverStream>; type ExecSandboxStream = ReceiverStream>; + type ConnectSupervisorStream = ReceiverStream>; async fn watch_sandbox( &self, @@ -301,6 +317,22 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { Err(Status::unimplemented("not implemented in test")) } + + async fn connect_supervisor( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + + type RelayStreamStream = ReceiverStream>; + + async fn relay_stream( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } } // --------------------------------------------------------------------------- @@ -420,9 +452,8 @@ where break; } } - Ok(Message::Close(_)) => break, + Ok(Message::Close(_)) | Err(_) => break, Ok(Message::Ping(_) | Message::Pong(_)) => {} - Err(_) => break, } } let _ = writer.shutdown().await; diff --git a/crates/openshell-tui/src/app.rs b/crates/openshell-tui/src/app.rs index 6a556fd124..1cab7127c2 100644 --- a/crates/openshell-tui/src/app.rs +++ b/crates/openshell-tui/src/app.rs @@ -359,7 +359,7 @@ pub struct CreateProviderForm { pub is_generic: bool, /// Status message (errors, validation). pub status: Option, - /// Warning shown at top of EnterKey modal (e.g. autodetect failure). + /// Warning shown at top of `EnterKey` modal (e.g. autodetect failure). pub warning: Option, /// Animation start time. pub anim_start: Option, @@ -463,6 +463,8 @@ pub struct App { pub sandbox_created: Vec, pub sandbox_images: Vec, pub sandbox_notes: Vec, + /// Formatted labels for each sandbox (e.g., "env=prod,team=platform" or empty string). + pub sandbox_labels: Vec, pub sandbox_policy_versions: Vec, pub sandbox_selected: usize, pub sandbox_count: usize, @@ -544,7 +546,39 @@ pub struct App { pub approve_all_confirm_chunks: Vec, } +// --------------------------------------------------------------------------- +// Label formatting utilities +// --------------------------------------------------------------------------- + +/// Sanitize a string for safe terminal display by filtering control characters. +/// +/// Removes all control characters except newlines to prevent ANSI escape +/// sequences or other terminal manipulation. +fn sanitize_for_display(s: &str) -> String { + s.chars() + .filter(|c| !c.is_control() || *c == '\n') + .collect() +} + +/// Format object labels as a comma-separated key=value string. +/// +/// Labels are sorted by key for deterministic output. Returns an empty string +/// if the map is empty. Values are sanitized to prevent terminal escape sequences. +pub fn format_labels(labels: &HashMap) -> String { + if labels.is_empty() { + return String::new(); + } + let mut sorted: Vec<_> = labels.iter().collect(); + sorted.sort_by_key(|(k, _)| *k); + sorted + .iter() + .map(|(k, v)| format!("{}={}", sanitize_for_display(k), sanitize_for_display(v))) + .collect::>() + .join(",") +} + impl App { + #[allow(clippy::large_types_passed_by_value)] // Theme is Copy; one-shot ctor pub fn new( client: OpenShellClient, gateway_name: String, @@ -597,6 +631,7 @@ impl App { sandbox_created: Vec::new(), sandbox_images: Vec::new(), sandbox_notes: Vec::new(), + sandbox_labels: Vec::new(), sandbox_policy_versions: Vec::new(), sandbox_selected: 0, sandbox_count: 0, @@ -685,18 +720,18 @@ impl App { self.sandbox_settings = settings::REGISTERED_SETTINGS .iter() .map(|reg| { - let (value, scope) = settings - .get(reg.key) - .map(|es| { - let v = es.value.as_ref().and_then(|sv| sv.value.clone()); - let s = match es.scope { - 1 => SettingScope::Sandbox, - 2 => SettingScope::Global, - _ => SettingScope::Unset, - }; - (v, s) - }) - .unwrap_or((None, SettingScope::Unset)); + let (value, scope) = + settings + .get(reg.key) + .map_or((None, SettingScope::Unset), |es| { + let v = es.value.as_ref().and_then(|sv| sv.value.clone()); + let s = match es.scope { + 1 => SettingScope::Sandbox, + 2 => SettingScope::Global, + _ => SettingScope::Unset, + }; + (v, s) + }); SandboxSettingEntry { key: reg.key.to_string(), kind: reg.kind, @@ -824,11 +859,8 @@ impl App { self.input_mode = InputMode::Command; self.command_input.clear(); } - KeyCode::Char('j') | KeyCode::Down => { - if !self.gateways.is_empty() { - self.gateway_selected = - (self.gateway_selected + 1).min(self.gateways.len() - 1); - } + KeyCode::Char('j') | KeyCode::Down if !self.gateways.is_empty() => { + self.gateway_selected = (self.gateway_selected + 1).min(self.gateways.len() - 1); } KeyCode::Char('k') | KeyCode::Up => { self.gateway_selected = self.gateway_selected.saturating_sub(1); @@ -868,11 +900,8 @@ impl App { self.input_mode = InputMode::Command; self.command_input.clear(); } - KeyCode::Char('j') | KeyCode::Down => { - if self.provider_count > 0 { - self.provider_selected = - (self.provider_selected + 1).min(self.provider_count - 1); - } + KeyCode::Char('j') | KeyCode::Down if self.provider_count > 0 => { + self.provider_selected = (self.provider_selected + 1).min(self.provider_count - 1); } KeyCode::Char('k') | KeyCode::Up => { self.provider_selected = self.provider_selected.saturating_sub(1); @@ -880,22 +909,16 @@ impl App { KeyCode::Char('c') => { self.open_create_provider_form(); } - KeyCode::Enter => { - // Fetch and show provider detail. - if self.provider_count > 0 { - self.pending_provider_get = true; - } + // Fetch and show provider detail. + KeyCode::Enter if self.provider_count > 0 => { + self.pending_provider_get = true; } - KeyCode::Char('u') => { - // Open update form for the selected provider. - if self.provider_count > 0 { - self.open_update_provider_form(); - } + // Open update form for the selected provider. + KeyCode::Char('u') if self.provider_count > 0 => { + self.open_update_provider_form(); } - KeyCode::Char('d') => { - if self.provider_count > 0 { - self.confirm_provider_delete = true; - } + KeyCode::Char('d') if self.provider_count > 0 => { + self.confirm_provider_delete = true; } KeyCode::Char('h' | 'l') | KeyCode::Left | KeyCode::Right => { self.middle_pane_tab = self.middle_pane_tab.next(); @@ -913,11 +936,9 @@ impl App { self.input_mode = InputMode::Command; self.command_input.clear(); } - KeyCode::Char('j') | KeyCode::Down => { - if !self.global_settings.is_empty() { - self.global_settings_selected = - (self.global_settings_selected + 1).min(self.global_settings.len() - 1); - } + KeyCode::Char('j') | KeyCode::Down if !self.global_settings.is_empty() => { + self.global_settings_selected = + (self.global_settings_selected + 1).min(self.global_settings.len() - 1); } KeyCode::Char('k') | KeyCode::Up => { self.global_settings_selected = self.global_settings_selected.saturating_sub(1); @@ -1048,10 +1069,8 @@ impl App { self.input_mode = InputMode::Command; self.command_input.clear(); } - KeyCode::Char('j') | KeyCode::Down => { - if self.sandbox_count > 0 { - self.sandbox_selected = (self.sandbox_selected + 1).min(self.sandbox_count - 1); - } + KeyCode::Char('j') | KeyCode::Down if self.sandbox_count > 0 => { + self.sandbox_selected = (self.sandbox_selected + 1).min(self.sandbox_count - 1); } KeyCode::Char('k') | KeyCode::Up => { self.sandbox_selected = self.sandbox_selected.saturating_sub(1); @@ -1059,13 +1078,11 @@ impl App { KeyCode::Char('c') => { self.open_create_form(); } - KeyCode::Enter => { - if self.sandbox_count > 0 { - self.screen = Screen::Sandbox; - self.focus = Focus::SandboxPolicy; - self.confirm_delete = false; - self.pending_sandbox_detail = true; - } + KeyCode::Enter if self.sandbox_count > 0 => { + self.screen = Screen::Sandbox; + self.focus = Focus::SandboxPolicy; + self.confirm_delete = false; + self.pending_sandbox_detail = true; } KeyCode::Esc => { self.focus = Focus::Providers; @@ -1116,10 +1133,8 @@ impl App { KeyCode::Char('r') => { self.focus = Focus::SandboxDraft; } - KeyCode::Char('s') => { - if self.sandbox_count > 0 { - self.pending_shell_connect = true; - } + KeyCode::Char('s') if self.sandbox_count > 0 => { + self.pending_shell_connect = true; } KeyCode::Char('d') => { self.confirm_delete = true; @@ -1161,11 +1176,9 @@ impl App { // In policy tab, 'l' opens logs. In settings tab, switch tab. self.sandbox_policy_tab = self.sandbox_policy_tab.next(); } - KeyCode::Char('j') | KeyCode::Down => { - if !self.sandbox_settings.is_empty() { - self.sandbox_settings_selected = - (self.sandbox_settings_selected + 1).min(self.sandbox_settings.len() - 1); - } + KeyCode::Char('j') | KeyCode::Down if !self.sandbox_settings.is_empty() => { + self.sandbox_settings_selected = + (self.sandbox_settings_selected + 1).min(self.sandbox_settings.len() - 1); } KeyCode::Char('k') | KeyCode::Up => { self.sandbox_settings_selected = self.sandbox_settings_selected.saturating_sub(1); @@ -1370,10 +1383,8 @@ impl App { self.focus = Focus::SandboxLogs; self.pending_log_fetch = true; } - KeyCode::Enter => { - if !self.draft_chunks.is_empty() { - self.draft_detail_open = true; - } + KeyCode::Enter if !self.draft_chunks.is_empty() => { + self.draft_detail_open = true; } KeyCode::Char('j') | KeyCode::Down => { if total == 0 { @@ -1401,13 +1412,11 @@ impl App { self.draft_scroll = 0; self.draft_selected = 0; } - KeyCode::Char('G') => { - if total > 0 { - let max_scroll = total.saturating_sub(vh.min(total)); - self.draft_scroll = max_scroll; - let visible = total.saturating_sub(self.draft_scroll).min(vh); - self.draft_selected = visible.saturating_sub(1); - } + KeyCode::Char('G') if total > 0 => { + let max_scroll = total.saturating_sub(vh.min(total)); + self.draft_scroll = max_scroll; + let visible = total.saturating_sub(self.draft_scroll).min(vh); + self.draft_selected = visible.saturating_sub(1); } // Approve selected chunk (pending → approved, rejected → approved). KeyCode::Char('a') => { @@ -1552,12 +1561,10 @@ impl App { self.log_autoscroll = false; } } - KeyCode::Enter => { - if filtered_len > 0 && self.log_selection_anchor.is_none() { - let abs = self.sandbox_log_scroll + self.log_cursor; - if abs < filtered_len { - self.log_detail_index = Some(abs); - } + KeyCode::Enter if filtered_len > 0 && self.log_selection_anchor.is_none() => { + let abs = self.sandbox_log_scroll + self.log_cursor; + if abs < filtered_len { + self.log_detail_index = Some(abs); } } KeyCode::Char('j') | KeyCode::Down => { @@ -1621,7 +1628,8 @@ impl App { self.sandbox_log_scroll = self.sandbox_log_scroll.saturating_sub(delta.unsigned_abs()); self.log_autoscroll = false; } else { - self.sandbox_log_scroll = (self.sandbox_log_scroll + delta as usize).min(max_scroll); + self.sandbox_log_scroll = + (self.sandbox_log_scroll + delta.cast_unsigned()).min(max_scroll); } let visible = filtered_len .saturating_sub(self.sandbox_log_scroll) @@ -1689,11 +1697,9 @@ impl App { CreateFormField::Image => Self::handle_text_input(&mut form.image, key), CreateFormField::Command => Self::handle_text_input(&mut form.command, key), CreateFormField::Providers => match key.code { - KeyCode::Char('j') | KeyCode::Down => { - if !form.providers.is_empty() { - form.provider_cursor = - (form.provider_cursor + 1).min(form.providers.len() - 1); - } + KeyCode::Char('j') | KeyCode::Down if !form.providers.is_empty() => { + form.provider_cursor = + (form.provider_cursor + 1).min(form.providers.len() - 1); } KeyCode::Char('k') | KeyCode::Up => { form.provider_cursor = form.provider_cursor.saturating_sub(1); @@ -1792,10 +1798,8 @@ impl App { KeyCode::Esc => { self.create_provider_form = None; } - KeyCode::Char('j') | KeyCode::Down => { - if !form.types.is_empty() { - form.type_cursor = (form.type_cursor + 1).min(form.types.len() - 1); - } + KeyCode::Char('j') | KeyCode::Down if !form.types.is_empty() => { + form.type_cursor = (form.type_cursor + 1).min(form.types.len() - 1); } KeyCode::Char('k') | KeyCode::Up => { form.type_cursor = form.type_cursor.saturating_sub(1); @@ -1836,7 +1840,7 @@ impl App { form.status = None; form.warning = None; } - KeyCode::Char('j') | KeyCode::Down | KeyCode::Char('k') | KeyCode::Up => { + KeyCode::Char('j' | 'k') | KeyCode::Down | KeyCode::Up => { form.method_cursor = 1 - form.method_cursor; } KeyCode::Enter => { @@ -2042,7 +2046,7 @@ impl App { registry .credential_env_vars(&ptype) .first() - .map_or(String::new(), |s| s.to_string()) + .map_or(String::new(), ToString::to_string) } else { cred_key }; @@ -2182,6 +2186,7 @@ impl App { self.sandbox_created.clear(); self.sandbox_images.clear(); self.sandbox_notes.clear(); + self.sandbox_labels.clear(); self.sandbox_policy_versions.clear(); self.sandbox_selected = 0; self.sandbox_count = 0; diff --git a/crates/openshell-tui/src/event.rs b/crates/openshell-tui/src/event.rs index e73862eb87..4c6eeb8b46 100644 --- a/crates/openshell-tui/src/event.rs +++ b/crates/openshell-tui/src/event.rs @@ -56,7 +56,7 @@ pub enum Event { pub struct EventHandler { rx: mpsc::UnboundedReceiver, // Kept alive so the spawned task's `tx` doesn't see a closed channel. - _keepalive: mpsc::UnboundedSender, + keepalive: mpsc::UnboundedSender, /// When true, the background reader stops polling stdin. paused: Arc, } @@ -85,20 +85,14 @@ impl EventHandler { if event::poll(poll_interval).unwrap_or(false) { match event::read() { - Ok(TermEvent::Key(key)) => { - if tx.send(Event::Key(key)).is_err() { - return; - } + Ok(TermEvent::Key(key)) if tx.send(Event::Key(key)).is_err() => { + return; } - Ok(TermEvent::Mouse(mouse)) => { - if tx.send(Event::Mouse(mouse)).is_err() { - return; - } + Ok(TermEvent::Mouse(mouse)) if tx.send(Event::Mouse(mouse)).is_err() => { + return; } - Ok(TermEvent::Resize(w, h)) => { - if tx.send(Event::Resize(w, h)).is_err() { - return; - } + Ok(TermEvent::Resize(w, h)) if tx.send(Event::Resize(w, h)).is_err() => { + return; } _ => {} } @@ -113,7 +107,7 @@ impl EventHandler { Self { rx, - _keepalive: keepalive, + keepalive, paused, } } @@ -124,7 +118,7 @@ impl EventHandler { /// Get a sender handle for dispatching events from background tasks. pub fn sender(&self) -> mpsc::UnboundedSender { - self._keepalive.clone() + self.keepalive.clone() } /// Pause stdin polling (call before suspending TUI for a child process). diff --git a/crates/openshell-tui/src/lib.rs b/crates/openshell-tui/src/lib.rs index f187f59fb4..8571ebbe1e 100644 --- a/crates/openshell-tui/src/lib.rs +++ b/crates/openshell-tui/src/lib.rs @@ -7,6 +7,7 @@ mod event; pub mod theme; mod ui; +use std::collections::HashMap; use std::io; use std::path::PathBuf; use std::time::Duration; @@ -17,6 +18,7 @@ use crossterm::terminal::{ EnterAlternateScreen, LeaveAlternateScreen, disable_raw_mode, enable_raw_mode, }; use miette::{IntoDiagnostic, Result}; +use openshell_core::metadata::{ObjectId, ObjectLabels, ObjectName}; use openshell_core::proto::open_shell_client::OpenShellClient; use ratatui::Terminal; use ratatui::backend::CrosstermBackend; @@ -32,9 +34,9 @@ const SPLASH_DURATION: Duration = Duration::from_secs(3); // Re-export for use by the CLI crate. pub use theme::ThemeMode; -/// Launch the OpenShell TUI. +/// Launch the `OpenShell` TUI. /// -/// `channel` must be a connected gRPC channel to the OpenShell gateway. +/// `channel` must be a connected gRPC channel to the `OpenShell` gateway. /// `theme_mode` selects the color theme: `Auto` detects the terminal /// background, `Dark`/`Light` forces a specific palette. pub async fn run( @@ -187,13 +189,13 @@ pub async fn run( .next() .cloned() .unwrap_or_default(); - let masked = if let Some(val) = provider.credentials.values().next() { - mask_secret(val) - } else { - "-".to_string() - }; + let masked = provider + .credentials + .values() + .next() + .map_or_else(|| "-".to_string(), |val| mask_secret(val)); app.provider_detail = Some(app::ProviderDetailView { - name: provider.name.clone(), + name: provider.object_name().to_string(), provider_type: provider.r#type.clone(), credential_key: cred_key, masked_value: masked, @@ -312,12 +314,11 @@ pub async fn run( }, Some(Event::Tick) => { // Auto-dismiss splash after SPLASH_DURATION. - if app.screen == Screen::Splash { - if let Some(start) = app.splash_start { - if start.elapsed() >= SPLASH_DURATION { - app.dismiss_splash(); - } - } + if app.screen == Screen::Splash + && let Some(start) = app.splash_start + && start.elapsed() >= SPLASH_DURATION + { + app.dismiss_splash(); } refresh_gateway_list(&mut app); @@ -337,90 +338,90 @@ pub async fn run( } Some(Event::Redraw) => { // Check if a buffered sandbox CreateResult is ready to finalize. - if let Some(form) = app.create_form.as_ref() { - if form.create_result.is_some() { - let elapsed = form - .anim_start - .map_or(app::MIN_CREATING_DISPLAY, |s| s.elapsed()); - if elapsed >= app::MIN_CREATING_DISPLAY { - let result = app - .create_form - .as_mut() - .and_then(|f| f.create_result.take()); - if let Some(h) = app.anim_handle.take() { - h.abort(); - } - match result { - Some(Ok(name)) => { - app.create_form = None; - let ports = std::mem::take(&mut app.pending_forward_ports); - let command = std::mem::take(&mut app.pending_exec_command); - let port_info = if ports.is_empty() { - String::new() - } else { - let list = ports - .iter() - .map(|p| p.to_string()) - .collect::>() - .join(", "); - format!(" (forwarding port(s) {list})") - }; - app.status_text = format!("Created sandbox: {name}{port_info}"); - refresh_sandboxes(&mut app).await; - - // If a command was specified, suspend TUI and exec it. - if !command.is_empty() { - handle_exec_command( - &mut app, - &mut terminal, - &events, - &name, - &command, - ) - .await; - } + if let Some(form) = app.create_form.as_ref() + && form.create_result.is_some() + { + let elapsed = form + .anim_start + .map_or(app::MIN_CREATING_DISPLAY, |s| s.elapsed()); + if elapsed >= app::MIN_CREATING_DISPLAY { + let result = app + .create_form + .as_mut() + .and_then(|f| f.create_result.take()); + if let Some(h) = app.anim_handle.take() { + h.abort(); + } + match result { + Some(Ok(name)) => { + app.create_form = None; + let ports = std::mem::take(&mut app.pending_forward_ports); + let command = std::mem::take(&mut app.pending_exec_command); + let port_info = if ports.is_empty() { + String::new() + } else { + let list = ports + .iter() + .map(ToString::to_string) + .collect::>() + .join(", "); + format!(" (forwarding port(s) {list})") + }; + app.status_text = format!("Created sandbox: {name}{port_info}"); + refresh_sandboxes(&mut app).await; + + // If a command was specified, suspend TUI and exec it. + if !command.is_empty() { + handle_exec_command( + &mut app, + &mut terminal, + &events, + &name, + &command, + ) + .await; } - Some(Err(msg)) => { - if let Some(form) = app.create_form.as_mut() { - form.phase = app::CreatePhase::Form; - form.anim_start = None; - form.status = Some(format!("Create failed: {msg}")); - } + } + Some(Err(msg)) => { + if let Some(form) = app.create_form.as_mut() { + form.phase = app::CreatePhase::Form; + form.anim_start = None; + form.status = Some(format!("Create failed: {msg}")); } - None => {} } + None => {} } } } // Check if a buffered provider CreateResult is ready to finalize. - if let Some(form) = app.create_provider_form.as_ref() { - if form.create_result.is_some() { - let elapsed = form - .anim_start - .map_or(app::MIN_CREATING_DISPLAY, |s| s.elapsed()); - if elapsed >= app::MIN_CREATING_DISPLAY { - let result = app - .create_provider_form - .as_mut() - .and_then(|f| f.create_result.take()); - if let Some(h) = app.anim_handle.take() { - h.abort(); + if let Some(form) = app.create_provider_form.as_ref() + && form.create_result.is_some() + { + let elapsed = form + .anim_start + .map_or(app::MIN_CREATING_DISPLAY, |s| s.elapsed()); + if elapsed >= app::MIN_CREATING_DISPLAY { + let result = app + .create_provider_form + .as_mut() + .and_then(|f| f.create_result.take()); + if let Some(h) = app.anim_handle.take() { + h.abort(); + } + match result { + Some(Ok(name)) => { + app.create_provider_form = None; + app.status_text = format!("Created provider: {name}"); + refresh_providers(&mut app).await; } - match result { - Some(Ok(name)) => { - app.create_provider_form = None; - app.status_text = format!("Created provider: {name}"); - refresh_providers(&mut app).await; + Some(Err(msg)) => { + if let Some(form) = app.create_provider_form.as_mut() { + form.phase = app::CreateProviderPhase::EnterKey; + form.anim_start = None; + form.status = Some(format!("Create failed: {msg}")); } - Some(Err(msg)) => { - if let Some(form) = app.create_provider_form.as_mut() { - form.phase = app::CreateProviderPhase::EnterKey; - form.anim_start = None; - form.status = Some(format!("Create failed: {msg}")); - } - } - None => {} } + None => {} } } } @@ -593,7 +594,7 @@ fn spawn_log_stream(app: &mut App, tx: mpsc::UnboundedSender) { source: String::new(), target: String::new(), message: format!("Failed to fetch logs: {}", e.message()), - fields: Default::default(), + fields: HashMap::default(), }])); return; } @@ -604,7 +605,7 @@ fn spawn_log_stream(app: &mut App, tx: mpsc::UnboundedSender) { source: String::new(), target: String::new(), message: "Timed out fetching logs.".into(), - fields: Default::default(), + fields: HashMap::default(), }])); return; } @@ -620,24 +621,20 @@ fn spawn_log_stream(app: &mut App, tx: mpsc::UnboundedSender) { ..Default::default() }; - let resp = - match tokio::time::timeout(Duration::from_secs(5), client.watch_sandbox(req)).await { - Ok(Ok(r)) => r, - Ok(Err(_)) | Err(_) => return, // Silently stop — user can re-enter logs. - }; + // Silently stop — user can re-enter logs. + let Ok(Ok(resp)) = + tokio::time::timeout(Duration::from_secs(5), client.watch_sandbox(req)).await + else { + return; + }; let mut stream = resp.into_inner(); - loop { - match stream.message().await { - Ok(Some(event)) => { - if let Some(openshell_core::proto::sandbox_stream_event::Payload::Log(log)) = - event.payload - { - let line = proto_to_log_line(log); - let _ = tx.send(Event::LogLines(vec![line])); - } - } - _ => break, // Stream ended or error. + while let Ok(Some(event)) = stream.message().await { + if let Some(openshell_core::proto::sandbox_stream_event::Payload::Log(log)) = + event.payload + { + let line = proto_to_log_line(log); + let _ = tx.send(Event::LogLines(vec![line])); } } }); @@ -716,13 +713,10 @@ async fn fetch_sandbox_detail(app: &mut App) { Ok(Ok(resp)) => { if let Some(sandbox) = resp.into_inner().sandbox { if let Some(spec) = &sandbox.spec { - app.sandbox_providers_list = spec.providers.clone(); - } - if sandbox.id.is_empty() { - None - } else { - Some(sandbox.id) + app.sandbox_providers_list.clone_from(&spec.providers); } + let id = sandbox.object_id().to_string(); + if id.is_empty() { None } else { Some(id) } } else { None } @@ -798,13 +792,14 @@ async fn handle_shell_connect( name: sandbox_name.clone(), }; match tokio::time::timeout(Duration::from_secs(5), app.client.get_sandbox(req)).await { - Ok(Ok(resp)) => match resp.into_inner().sandbox { - Some(s) => s.id, - None => { + Ok(Ok(resp)) => { + if let Some(s) = resp.into_inner().sandbox { + s.object_id().to_string() + } else { app.status_text = "sandbox not found".to_string(); return; } - }, + } Ok(Err(e)) => { app.status_text = format!("failed to get sandbox: {}", e.message()); return; @@ -834,6 +829,10 @@ async fn handle_shell_connect( } } }; + if let Err(err) = validate_ssh_session_response(&session) { + app.status_text = format!("gateway returned invalid SSH session response: {err}"); + return; + } // Step 3: Resolve gateway address (handle loopback override). #[allow(clippy::cast_possible_truncation)] @@ -853,11 +852,12 @@ async fn handle_shell_connect( return; } }; - let exe_str = shell_escape(&exe.to_string_lossy()); - let gateway = shell_escape(&app.gateway_name); - let proxy_command = format!( - "{exe_str} ssh-proxy --gateway {gateway_url} --sandbox-id {} --token {} --gateway-name {gateway}", - session.sandbox_id, session.token, + let proxy_command = build_proxy_command( + &exe.to_string_lossy(), + &gateway_url, + &session.sandbox_id, + &session.token, + &app.gateway_name, ); // Step 5: Build the SSH command. let mut command = std::process::Command::new("ssh"); @@ -942,13 +942,14 @@ async fn handle_exec_command( name: sandbox_name.to_string(), }; match tokio::time::timeout(Duration::from_secs(5), app.client.get_sandbox(req)).await { - Ok(Ok(resp)) => match resp.into_inner().sandbox { - Some(s) => s.id, - None => { + Ok(Ok(resp)) => { + if let Some(s) = resp.into_inner().sandbox { + s.object_id().to_string() + } else { app.status_text = format!("exec: sandbox {sandbox_name} not found"); return; } - }, + } Ok(Err(e)) => { app.status_text = format!("exec: failed to get sandbox: {}", e.message()); return; @@ -977,6 +978,10 @@ async fn handle_exec_command( } } }; + if let Err(err) = validate_ssh_session_response(&session) { + app.status_text = format!("exec: gateway returned invalid SSH session response: {err}"); + return; + } // Step 2: Resolve gateway and build ProxyCommand (same as handle_shell_connect). #[allow(clippy::cast_possible_truncation)] @@ -995,11 +1000,12 @@ async fn handle_exec_command( return; } }; - let exe_str = shell_escape(&exe.to_string_lossy()); - let gateway = shell_escape(&app.gateway_name); - let proxy_command = format!( - "{exe_str} ssh-proxy --gateway {gateway_url} --sandbox-id {} --token {} --gateway-name {gateway}", - session.sandbox_id, session.token, + let proxy_command = build_proxy_command( + &exe.to_string_lossy(), + &gateway_url, + &session.sandbox_id, + &session.token, + &app.gateway_name, ); // Step 3: Build SSH command — same flags as handle_shell_connect but with @@ -1007,7 +1013,7 @@ async fn handle_exec_command( // remote shell parses it correctly. let command_str = command .split_whitespace() - .map(|word| shell_escape(word)) + .map(shell_escape) .collect::>() .join(" "); let mut ssh = std::process::Command::new("ssh"); @@ -1073,7 +1079,9 @@ async fn handle_exec_command( } // SSH utility functions are shared via openshell_core::forward. -use openshell_core::forward::{resolve_ssh_gateway, shell_escape}; +use openshell_core::forward::{ + build_proxy_command, resolve_ssh_gateway, shell_escape, validate_ssh_session_response, +}; /// Convert a `SandboxPolicy` proto into styled ratatui lines for the policy viewer. fn render_policy_lines( @@ -1197,7 +1205,7 @@ fn render_policy_lines( }; lines.push(Line::from(vec![ Span::styled(" Allow: ", t.muted), - Span::styled(format!("{:<6} {}", method, target), t.text), + Span::styled(format!("{method:<6} {target}"), t.text), ])); } } @@ -1265,7 +1273,7 @@ fn spawn_create_sandbox(app: &mut App, tx: mpsc::UnboundedSender) { // Stash command so we can exec after sandbox creation + Ready. app.pending_exec_command = command; // Stash ports so we can include them in the status text. - app.pending_forward_ports = ports.clone(); + app.pending_forward_ports.clone_from(&ports); let endpoint = app.endpoint.clone(); let gateway_name = app.gateway_name.clone(); @@ -1301,14 +1309,22 @@ fn spawn_create_sandbox(app: &mut App, tx: mpsc::UnboundedSender) { policy, ..Default::default() }), + labels: HashMap::new(), }; let sandbox_name = match tokio::time::timeout(Duration::from_secs(30), client.create_sandbox(req)).await { - Ok(Ok(resp)) => resp - .into_inner() - .sandbox - .map_or_else(|| "unknown".to_string(), |s| s.name), + Ok(Ok(resp)) => resp.into_inner().sandbox.map_or_else( + || "unknown".to_string(), + |s| { + let name = s.object_name().to_string(); + if name.is_empty() { + "unknown".to_string() + } else { + name + } + }, + ), Ok(Err(e)) => { let _ = tx.send(Event::CreateResult(Err(e.message().to_string()))); return; @@ -1335,21 +1351,19 @@ fn spawn_create_sandbox(app: &mut App, tx: mpsc::UnboundedSender) { let req = openshell_core::proto::GetSandboxRequest { name: sandbox_name.clone(), }; - match client.get_sandbox(req).await { - Ok(resp) => { - if let Some(sandbox) = resp.into_inner().sandbox { - if sandbox.phase == 2 { - break sandbox.id; - } - if sandbox.phase == 3 { - let _ = tx.send(Event::CreateResult(Err( - "sandbox entered error state".to_string(), - ))); - return; - } - } + // Retry on transient errors. + if let Ok(resp) = client.get_sandbox(req).await + && let Some(sandbox) = resp.into_inner().sandbox + { + if sandbox.phase == 2 { + break sandbox.object_id().to_string(); + } + if sandbox.phase == 3 { + let _ = tx.send(Event::CreateResult(Err( + "sandbox entered error state".to_string() + ))); + return; } - Err(_) => {} // Retry on transient errors. } }; @@ -1400,6 +1414,10 @@ async fn start_port_forwards( } } }; + if let Err(err) = validate_ssh_session_response(&session) { + tracing::warn!("gateway returned invalid SSH session response for forwards: {err}"); + return; + } // Resolve gateway address. #[allow(clippy::cast_possible_truncation)] @@ -1419,11 +1437,12 @@ async fn start_port_forwards( return; } }; - let exe_str = shell_escape(&exe.to_string_lossy()); - let gateway = shell_escape(gateway_name); - let proxy_command = format!( - "{exe_str} ssh-proxy --gateway {gateway_url} --sandbox-id {} --token {} --gateway-name {gateway}", - session.sandbox_id, session.token, + let proxy_command = build_proxy_command( + &exe.to_string_lossy(), + &gateway_url, + &session.sandbox_id, + &session.token, + gateway_name, ); // Start a forward for each spec. @@ -1541,17 +1560,28 @@ fn spawn_create_provider(app: &App, tx: mpsc::UnboundedSender) { let req = openshell_core::proto::CreateProviderRequest { provider: Some(openshell_core::proto::Provider { - id: String::new(), - name: provider_name.clone(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: provider_name.clone(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: ptype.clone(), credentials: credentials.clone(), - config: Default::default(), + config: HashMap::default(), }), }; match client.create_provider(req).await { Ok(resp) => { - let final_name = resp.into_inner().provider.map_or(provider_name, |p| p.name); + let final_name = resp.into_inner().provider.map_or(provider_name, |p| { + let name = p.object_name().to_string(); + if name.is_empty() { + "unknown".to_string() + } else { + name + } + }); let _ = tx.send(Event::ProviderCreateResult(Ok(final_name))); return; } @@ -1615,16 +1645,20 @@ fn spawn_update_provider(app: &App, tx: mpsc::UnboundedSender) { let new_value = form.new_value.clone(); tokio::spawn(async move { - let mut credentials = std::collections::HashMap::new(); + let mut credentials = HashMap::new(); credentials.insert(cred_key, new_value); let req = openshell_core::proto::UpdateProviderRequest { provider: Some(openshell_core::proto::Provider { - id: String::new(), - name: name.clone(), + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.clone(), + created_at_ms: 0, + labels: HashMap::new(), + }), r#type: ptype, credentials, - config: Default::default(), + config: HashMap::default(), }), }; @@ -1848,7 +1882,10 @@ async fn refresh_providers(app: &mut App) { Ok(Ok(resp)) => { let providers = resp.into_inner().providers; app.provider_count = providers.len(); - app.provider_names = providers.iter().map(|p| p.name.clone()).collect(); + app.provider_names = providers + .iter() + .map(|p| p.object_name().to_string()) + .collect(); app.provider_types = providers.iter().map(|p| p.r#type.clone()).collect(); app.provider_cred_keys = providers .iter() @@ -1929,25 +1966,25 @@ fn spawn_set_global_setting(app: &App, tx: mpsc::UnboundedSender) { let value = match kind { openshell_core::settings::SettingValueKind::Bool => { - match openshell_core::settings::parse_bool_like(&raw) { - Some(v) => setting_value::Value::BoolValue(v), - None => { - let _ = tx.send(Event::GlobalSettingSetResult(Err(format!( - "invalid bool value: {raw}" - )))); - return; - } + if let Some(v) = openshell_core::settings::parse_bool_like(&raw) { + setting_value::Value::BoolValue(v) + } else { + let _ = tx.send(Event::GlobalSettingSetResult(Err(format!( + "invalid bool value: {raw}" + )))); + return; } } - openshell_core::settings::SettingValueKind::Int => match raw.parse::() { - Ok(v) => setting_value::Value::IntValue(v), - Err(_) => { + openshell_core::settings::SettingValueKind::Int => { + if let Ok(v) = raw.parse::() { + setting_value::Value::IntValue(v) + } else { let _ = tx.send(Event::GlobalSettingSetResult(Err(format!( "invalid int value: {raw}" )))); return; } - }, + } openshell_core::settings::SettingValueKind::String => { setting_value::Value::StringValue(raw) } @@ -1960,6 +1997,7 @@ fn spawn_set_global_setting(app: &App, tx: mpsc::UnboundedSender) { setting_value: Some(SettingValue { value: Some(value) }), delete_setting: false, global: true, + merge_operations: vec![], }; let result = tokio::time::timeout(Duration::from_secs(5), client.update_config(req)).await; @@ -1994,6 +2032,7 @@ fn spawn_delete_global_setting(app: &App, tx: mpsc::UnboundedSender) { setting_value: None, delete_setting: true, global: true, + merge_operations: vec![], }; let result = tokio::time::timeout(Duration::from_secs(5), client.update_config(req)).await; @@ -2031,25 +2070,25 @@ fn spawn_set_sandbox_setting(app: &App, tx: mpsc::UnboundedSender) { let value = match kind { openshell_core::settings::SettingValueKind::Bool => { - match openshell_core::settings::parse_bool_like(&raw) { - Some(v) => setting_value::Value::BoolValue(v), - None => { - let _ = tx.send(Event::SandboxSettingSetResult(Err(format!( - "invalid bool value: {raw}" - )))); - return; - } + if let Some(v) = openshell_core::settings::parse_bool_like(&raw) { + setting_value::Value::BoolValue(v) + } else { + let _ = tx.send(Event::SandboxSettingSetResult(Err(format!( + "invalid bool value: {raw}" + )))); + return; } } - openshell_core::settings::SettingValueKind::Int => match raw.parse::() { - Ok(v) => setting_value::Value::IntValue(v), - Err(_) => { + openshell_core::settings::SettingValueKind::Int => { + if let Ok(v) = raw.parse::() { + setting_value::Value::IntValue(v) + } else { let _ = tx.send(Event::SandboxSettingSetResult(Err(format!( "invalid int value: {raw}" )))); return; } - }, + } openshell_core::settings::SettingValueKind::String => { setting_value::Value::StringValue(raw) } @@ -2062,6 +2101,7 @@ fn spawn_set_sandbox_setting(app: &App, tx: mpsc::UnboundedSender) { setting_value: Some(SettingValue { value: Some(value) }), delete_setting: false, global: false, + merge_operations: vec![], }; let result = tokio::time::timeout(Duration::from_secs(5), client.update_config(req)).await; @@ -2100,6 +2140,7 @@ fn spawn_delete_sandbox_setting(app: &App, tx: mpsc::UnboundedSender) { setting_value: None, delete_setting: true, global: false, + merge_operations: vec![], }; let result = tokio::time::timeout(Duration::from_secs(5), client.update_config(req)).await; @@ -2141,6 +2182,7 @@ async fn refresh_sandboxes(app: &mut App) { let req = openshell_core::proto::ListSandboxesRequest { limit: 100, offset: 0, + label_selector: String::new(), }; let result = tokio::time::timeout(Duration::from_secs(5), app.client.list_sandboxes(req)).await; match result { @@ -2153,8 +2195,14 @@ async fn refresh_sandboxes(app: &mut App) { Ok(Ok(resp)) => { let sandboxes = resp.into_inner().sandboxes; app.sandbox_count = sandboxes.len(); - app.sandbox_ids = sandboxes.iter().map(|s| s.id.clone()).collect(); - app.sandbox_names = sandboxes.iter().map(|s| s.name.clone()).collect(); + app.sandbox_ids = sandboxes + .iter() + .map(|s| s.object_id().to_string()) + .collect(); + app.sandbox_names = sandboxes + .iter() + .map(|s| s.object_name().to_string()) + .collect(); app.sandbox_phases = sandboxes.iter().map(|s| phase_label(s.phase)).collect(); app.sandbox_images = sandboxes .iter() @@ -2170,11 +2218,19 @@ async fn refresh_sandboxes(app: &mut App) { .collect(); app.sandbox_ages = sandboxes .iter() - .map(|s| format_age(s.created_at_ms)) + .map(|s| { + s.metadata + .as_ref() + .map_or_else(|| "?".to_string(), |m| format_age(m.created_at_ms)) + }) .collect(); app.sandbox_created = sandboxes .iter() - .map(|s| format_timestamp(s.created_at_ms)) + .map(|s| { + s.metadata + .as_ref() + .map_or_else(|| "?".to_string(), |m| format_timestamp(m.created_at_ms)) + }) .collect(); app.sandbox_policy_versions = @@ -2184,7 +2240,21 @@ async fn refresh_sandboxes(app: &mut App) { let forwards = openshell_core::forward::list_forwards().unwrap_or_default(); app.sandbox_notes = sandboxes .iter() - .map(|s| openshell_core::forward::build_sandbox_notes(&s.name, &forwards)) + .map(|s| { + let name = s.object_name(); + openshell_core::forward::build_sandbox_notes(name, &forwards) + }) + .collect(); + + // Build LABELS column from metadata. + app.sandbox_labels = sandboxes + .iter() + .map(|s| { + s.object_labels() + .as_ref() + .map(app::format_labels) + .unwrap_or_default() + }) .collect(); if app.sandbox_selected >= app.sandbox_count && app.sandbox_count > 0 { diff --git a/crates/openshell-tui/src/theme.rs b/crates/openshell-tui/src/theme.rs index 1be2023671..0c0d9b37c3 100644 --- a/crates/openshell-tui/src/theme.rs +++ b/crates/openshell-tui/src/theme.rs @@ -213,7 +213,7 @@ pub fn detect(mode: ThemeMode) -> Theme { /// /// Uses `terminal-colorsaurus` to send an OSC 11 query to the terminal, /// which returns the actual background RGB color. This works reliably on -/// iTerm2, Terminal.app, WezTerm, Alacritty, and most modern terminals. +/// iTerm2, Terminal.app, `WezTerm`, Alacritty, and most modern terminals. /// /// Falls back to `false` (dark) if the terminal doesn't respond to the /// query (e.g. `TERM=dumb`, piped output, very old terminals). diff --git a/crates/openshell-tui/src/ui/create_provider.rs b/crates/openshell-tui/src/ui/create_provider.rs index c099ea6a57..9f1cc6d832 100644 --- a/crates/openshell-tui/src/ui/create_provider.rs +++ b/crates/openshell-tui/src/ui/create_provider.rs @@ -313,7 +313,6 @@ fn draw_enter_key( chunks[idx], t, ); - idx += 1; } else { // Credential rows — env var name + masked value on the same line. let max_name_len = form @@ -330,7 +329,7 @@ fn draw_enter_key( .map(|(i, (env_name, value))| { let is_focused = form.key_field == ProviderKeyField::Credential && i == form.cred_cursor; - let padded = format!("{:width$}", env_name, width = max_name_len); + let padded = format!("{env_name:max_name_len$}"); let name_style = if is_focused { t.accent_bold } else { t.text }; let mut spans = vec![Span::styled(format!(" {padded}: "), name_style)]; if value.is_empty() { @@ -353,8 +352,8 @@ fn draw_enter_key( }) .collect(); frame.render_widget(Paragraph::new(lines), chunks[idx]); - idx += 1; } + idx += 1; // Spacer. idx += 1; diff --git a/crates/openshell-tui/src/ui/create_sandbox.rs b/crates/openshell-tui/src/ui/create_sandbox.rs index c8411c6184..cb90e244de 100644 --- a/crates/openshell-tui/src/ui/create_sandbox.rs +++ b/crates/openshell-tui/src/ui/create_sandbox.rs @@ -304,7 +304,7 @@ pub fn render_chase( } let frame = (elapsed_ms / 140) as usize; - let mouth_open = frame % 2 == 0; + let mouth_open = frame.is_multiple_of(2); // Characters. let pac = if mouth_open { "ᗧ" } else { "●" }; @@ -360,16 +360,14 @@ pub fn render_chase( let mut current_style = buf[0].1; for &(ch, style) in &buf { - if style == current_style { - current_str.push(ch); - } else { + if style != current_style { if !current_str.is_empty() { spans.push(Span::styled(current_str.clone(), current_style)); current_str.clear(); } current_style = style; - current_str.push(ch); } + current_str.push(ch); } if !current_str.is_empty() { spans.push(Span::styled(current_str, current_style)); diff --git a/crates/openshell-tui/src/ui/global_settings.rs b/crates/openshell-tui/src/ui/global_settings.rs index cac59b0a9f..9e47fd8d62 100644 --- a/crates/openshell-tui/src/ui/global_settings.rs +++ b/crates/openshell-tui/src/ui/global_settings.rs @@ -155,7 +155,7 @@ fn draw_edit_overlay( ])); // content lines + 2 for border - let popup_height = (lines.len() + 2) as u16; + let popup_height = u16::try_from(lines.len() + 2).unwrap_or(u16::MAX); let popup = centered_rect(50, popup_height, area); frame.render_widget(Clear, popup); @@ -239,7 +239,7 @@ fn draw_confirm_delete(frame: &mut Frame<'_>, app: &App, idx: usize, area: Rect) ]; // content lines + 2 for border - let popup_height = (lines.len() + 2) as u16; + let popup_height = u16::try_from(lines.len() + 2).unwrap_or(u16::MAX); let popup = centered_rect(60, popup_height, area); frame.render_widget(Clear, popup); diff --git a/crates/openshell-tui/src/ui/mod.rs b/crates/openshell-tui/src/ui/mod.rs index b920d9cb69..13ac94c10a 100644 --- a/crates/openshell-tui/src/ui/mod.rs +++ b/crates/openshell-tui/src/ui/mod.rs @@ -1,17 +1,17 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -pub(crate) mod create_provider; -pub(crate) mod create_sandbox; +pub mod create_provider; +pub mod create_sandbox; mod dashboard; -pub(crate) mod global_settings; -pub(crate) mod providers; -pub(crate) mod sandbox_detail; +pub mod global_settings; +pub mod providers; +pub mod sandbox_detail; mod sandbox_draft; -pub(crate) mod sandbox_logs; +pub mod sandbox_logs; mod sandbox_policy; -pub(crate) mod sandbox_settings; -pub(crate) mod sandboxes; +pub mod sandbox_settings; +pub mod sandboxes; mod splash; use ratatui::Frame; @@ -89,12 +89,12 @@ fn draw_sandbox_screen(frame: &mut Frame<'_>, app: &mut App, area: Rect) { } // Log detail popup renders over the full frame (not constrained to pane). - if app.focus == Focus::SandboxLogs { - if let Some(detail_idx) = app.log_detail_index { - let filtered: Vec<&app::LogLine> = app.filtered_log_lines(); - if let Some(log) = filtered.get(detail_idx) { - sandbox_logs::draw_detail_popup(frame, log, frame.size(), &app.theme); - } + if app.focus == Focus::SandboxLogs + && let Some(detail_idx) = app.log_detail_index + { + let filtered: Vec<&app::LogLine> = app.filtered_log_lines(); + if let Some(log) = filtered.get(detail_idx) { + sandbox_logs::draw_detail_popup(frame, log, frame.size(), &app.theme); } } @@ -328,8 +328,7 @@ fn draw_nav_bar(frame: &mut Frame<'_>, app: &App, area: Rect) { let selected_status = app .draft_chunks .get(app.draft_scroll + app.draft_selected) - .map(|c| c.status.as_str()) - .unwrap_or(""); + .map_or("", |c| c.status.as_str()); let mut spans = vec![ Span::styled(" ", t.text), Span::styled("[j/k]", t.key_hint), @@ -456,7 +455,7 @@ fn draw_command_bar(frame: &mut Frame<'_>, app: &App, area: Rect) { /// Center a popup rectangle within `area` using percentage-based width and /// an absolute height (in rows). -pub(crate) fn centered_popup(percent_x: u16, height: u16, area: Rect) -> Rect { +pub fn centered_popup(percent_x: u16, height: u16, area: Rect) -> Rect { let vert = Layout::default() .direction(Direction::Vertical) .constraints([ diff --git a/crates/openshell-tui/src/ui/sandbox_detail.rs b/crates/openshell-tui/src/ui/sandbox_detail.rs index 0eab117824..7cdbec8bd0 100644 --- a/crates/openshell-tui/src/ui/sandbox_detail.rs +++ b/crates/openshell-tui/src/ui/sandbox_detail.rs @@ -73,29 +73,40 @@ pub fn draw(frame: &mut Frame<'_>, app: &App, area: Rect) { Span::styled(age, t.text), ]); - // Row 3: Providers + // Row 3: Labels + let labels_str = app + .sandbox_labels + .get(idx) + .filter(|s| !s.is_empty()) + .map_or("none", String::as_str); + let row3 = Line::from(vec![ + Span::styled(" Labels: ", t.muted), + Span::styled(labels_str, t.text), + ]); + + // Row 4: Providers let providers_str = if app.sandbox_providers_list.is_empty() { "none".to_string() } else { app.sandbox_providers_list.join(", ") }; - let row3 = Line::from(vec![ + let row4 = Line::from(vec![ Span::styled(" Providers: ", t.muted), Span::styled(providers_str, t.text), ]); - // Row 4: Forwarded Ports + // Row 5: Forwarded Ports let forwards_str = app .sandbox_notes .get(idx) .filter(|s| !s.is_empty()) - .map_or_else(|| "none".to_string(), Clone::clone); - let row4 = Line::from(vec![ + .map_or("none", String::as_str); + let row5 = Line::from(vec![ Span::styled(" Forwards: ", t.muted), Span::styled(forwards_str, t.text), ]); - let mut lines = vec![Line::from(""), row1, row2, row3, row4]; + let mut lines = vec![Line::from(""), row1, row2, row3, row4, row5]; // Show global policy indicator when the sandbox's policy is managed at // gateway scope. diff --git a/crates/openshell-tui/src/ui/sandbox_draft.rs b/crates/openshell-tui/src/ui/sandbox_draft.rs index 528d1c6086..a9cd6b0c90 100644 --- a/crates/openshell-tui/src/ui/sandbox_draft.rs +++ b/crates/openshell-tui/src/ui/sandbox_draft.rs @@ -324,7 +324,7 @@ pub fn draw_approve_all_popup( let count = chunks.len(); // Height: header(1) + blank(1) + chunks(count, capped at 12) + blank(1) + hints(1) + borders(2) + padding(1) let list_lines = count.min(12); - let popup_height = (7 + list_lines) as u16; + let popup_height = u16::try_from(7 + list_lines).unwrap_or(u16::MAX); let popup_height = popup_height.min(area.height.saturating_sub(4)); let popup_width = (area.width * 4 / 5).min(area.width.saturating_sub(4)); let popup_area = centered_rect(popup_width, popup_height, area); diff --git a/crates/openshell-tui/src/ui/sandbox_logs.rs b/crates/openshell-tui/src/ui/sandbox_logs.rs index 2aa793d20e..da20b40c88 100644 --- a/crates/openshell-tui/src/ui/sandbox_logs.rs +++ b/crates/openshell-tui/src/ui/sandbox_logs.rs @@ -335,7 +335,7 @@ const CONNECT_FIELD_ORDER: &[&str] = &[ "reason", ]; -/// Priority field order for L7_REQUEST log lines. +/// Priority field order for `L7_REQUEST` log lines. const L7_FIELD_ORDER: &[&str] = &[ "l7_action", "l7_target", @@ -348,7 +348,7 @@ const L7_FIELD_ORDER: &[&str] = &[ ]; /// Return fields in a smart order based on the log message type. -pub(crate) fn ordered_fields<'a>(log: &'a LogLine) -> Vec<(&'a str, &'a str)> { +pub fn ordered_fields(log: &LogLine) -> Vec<(&str, &str)> { // Matches both "CONNECT" (L4-only decision) and "CONNECT_L7" (tunnel lifecycle for L7 endpoints) let order: Option<&[&str]> = if log.message.starts_with("CONNECT") { Some(CONNECT_FIELD_ORDER) @@ -358,8 +358,18 @@ pub(crate) fn ordered_fields<'a>(log: &'a LogLine) -> Vec<(&'a str, &'a str)> { None }; - match order { - Some(priority) => { + order.map_or_else( + || { + // Default: alphabetical. + let mut pairs: Vec<(&str, &str)> = log + .fields + .iter() + .map(|(k, v)| (k.as_str(), v.as_str())) + .collect(); + pairs.sort_by_key(|(k, _)| *k); + pairs + }, + |priority| { let mut result: Vec<(&str, &str)> = Vec::with_capacity(log.fields.len()); // Add priority fields first (in order). for &key in priority { @@ -377,18 +387,8 @@ pub(crate) fn ordered_fields<'a>(log: &'a LogLine) -> Vec<(&'a str, &'a str)> { remaining.sort_by_key(|(k, _)| *k); result.extend(remaining); result - } - None => { - // Default: alphabetical. - let mut pairs: Vec<(&str, &str)> = log - .fields - .iter() - .map(|(k, v)| (k.as_str(), v.as_str())) - .collect(); - pairs.sort_by_key(|(k, _)| *k); - pairs - } - } + }, + ) } // --------------------------------------------------------------------------- @@ -404,7 +404,7 @@ fn level_style(level: &str, t: &crate::theme::Theme) -> ratatui::style::Style { } } -pub(crate) fn format_short_time(epoch_ms: i64) -> String { +pub fn format_short_time(epoch_ms: i64) -> String { if epoch_ms <= 0 { return String::from("--:--:--"); } @@ -420,7 +420,7 @@ pub(crate) fn format_short_time(epoch_ms: i64) -> String { /// /// Produces the same layout as `render_log_line()` but without styles or /// truncation: `HH:MM:SS {source:<7} {level:<5} {message} [key=value ...]` -pub(crate) fn format_log_line_plain(log: &LogLine) -> String { +pub fn format_log_line_plain(log: &LogLine) -> String { let ts = format_short_time(log.timestamp_ms); let mut s = format!("{ts} {:<7} {:<5} {}", log.source, log.level, log.message); diff --git a/crates/openshell-tui/src/ui/sandbox_settings.rs b/crates/openshell-tui/src/ui/sandbox_settings.rs index c26f4a6632..8ac5eb687e 100644 --- a/crates/openshell-tui/src/ui/sandbox_settings.rs +++ b/crates/openshell-tui/src/ui/sandbox_settings.rs @@ -163,7 +163,7 @@ fn draw_edit_overlay( Span::styled(" Cancel", t.muted), ])); - let popup_height = (lines.len() + 2) as u16; + let popup_height = u16::try_from(lines.len() + 2).unwrap_or(u16::MAX); let popup = centered_rect(50, popup_height, area); frame.render_widget(Clear, popup); @@ -207,7 +207,7 @@ fn draw_confirm_set(frame: &mut Frame<'_>, app: &App, idx: usize, area: Rect) { ]), ]; - let popup_height = (lines.len() + 2) as u16; + let popup_height = u16::try_from(lines.len() + 2).unwrap_or(u16::MAX); let popup = centered_rect(60, popup_height, area); frame.render_widget(Clear, popup); @@ -245,7 +245,7 @@ fn draw_confirm_delete(frame: &mut Frame<'_>, app: &App, idx: usize, area: Rect) ]), ]; - let popup_height = (lines.len() + 2) as u16; + let popup_height = u16::try_from(lines.len() + 2).unwrap_or(u16::MAX); let popup = centered_rect(55, popup_height, area); frame.render_widget(Clear, popup); diff --git a/crates/openshell-tui/src/ui/sandboxes.rs b/crates/openshell-tui/src/ui/sandboxes.rs index 39f6411c43..3fb8e3a3b0 100644 --- a/crates/openshell-tui/src/ui/sandboxes.rs +++ b/crates/openshell-tui/src/ui/sandboxes.rs @@ -16,6 +16,7 @@ pub fn draw(frame: &mut Frame<'_>, app: &App, area: Rect, focused: bool) { Cell::from(Span::styled("CREATED", t.muted)), Cell::from(Span::styled("AGE", t.muted)), Cell::from(Span::styled("IMAGE", t.muted)), + Cell::from(Span::styled("LABELS", t.muted)), Cell::from(Span::styled("NOTES", t.muted)), ]) .bottom_margin(1); @@ -27,6 +28,7 @@ pub fn draw(frame: &mut Frame<'_>, app: &App, area: Rect, focused: bool) { let created = app.sandbox_created.get(i).map_or("", String::as_str); let age = app.sandbox_ages.get(i).map_or("", String::as_str); let image = app.sandbox_images.get(i).map_or("", String::as_str); + let labels = app.sandbox_labels.get(i).map_or("", String::as_str); let notes = app.sandbox_notes.get(i).map_or("", String::as_str); let draft_count = app.sandbox_draft_counts.get(i).copied().unwrap_or(0); @@ -64,6 +66,7 @@ pub fn draw(frame: &mut Frame<'_>, app: &App, area: Rect, focused: bool) { Cell::from(Span::styled(created, t.muted)), Cell::from(Span::styled(age, t.muted)), Cell::from(Span::styled(image, t.muted)), + Cell::from(Span::styled(labels, t.muted)), Cell::from(Span::styled(notes, t.muted)), ]) }) @@ -74,8 +77,9 @@ pub fn draw(frame: &mut Frame<'_>, app: &App, area: Rect, focused: bool) { Constraint::Percentage(10), Constraint::Percentage(15), Constraint::Percentage(8), - Constraint::Percentage(27), Constraint::Percentage(20), + Constraint::Percentage(15), + Constraint::Percentage(12), ]; let border_style = if focused { t.border_focused } else { t.border }; diff --git a/crates/openshell-vfio/Cargo.toml b/crates/openshell-vfio/Cargo.toml new file mode 100644 index 0000000000..b93500fdcb --- /dev/null +++ b/crates/openshell-vfio/Cargo.toml @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-vfio" +description = "VFIO GPU passthrough lifecycle for OpenShell VM sandboxes" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +serde = { workspace = true } +serde_json = { workspace = true } +thiserror = { workspace = true } +tracing = { workspace = true } + +[dev-dependencies] +tempfile = "3" + +[lints] +workspace = true diff --git a/crates/openshell-vfio/src/lib.rs b/crates/openshell-vfio/src/lib.rs new file mode 100644 index 0000000000..30235b1e3f --- /dev/null +++ b/crates/openshell-vfio/src/lib.rs @@ -0,0 +1,1599 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! VFIO GPU passthrough lifecycle management for `OpenShell` VM sandboxes. +//! +//! Provides discovery, binding, and crash-recovery for NVIDIA GPUs using +//! the VFIO subsystem. All sysfs access goes through [`SysfsRoot`] so the +//! entire stack is testable without root or real hardware. + +use std::collections::HashMap; +use std::fs; +use std::path::{Path, PathBuf}; +use std::sync::Mutex; +use std::time::Duration; + +const NVIDIA_VENDOR_ID: &str = "0x10de"; +const GPU_CLASS_DISPLAY_VGA: &str = "0x030000"; +const GPU_CLASS_DISPLAY_3D: u32 = 0x0302; + +const VFIO_BIND_POLL_INTERVAL: Duration = Duration::from_millis(100); +const VFIO_BIND_MAX_POLL_ATTEMPTS: u32 = 20; + +/// Reference counter for vendor:device ID registrations in the vfio-pci +/// match table. Multiple GPUs may share the same vendor:device pair (e.g., +/// two A100s). We only write to the kernel's `new_id`/`remove_id` sysfs +/// files when the first GPU registers or the last GPU deregisters an ID. +static VFIO_ID_REFCOUNTS: std::sync::LazyLock>> = + std::sync::LazyLock::new(|| Mutex::new(HashMap::new())); + +// --------------------------------------------------------------------------- +// Errors +// --------------------------------------------------------------------------- + +#[derive(Debug, thiserror::Error)] +pub enum VfioError { + #[error("GPU {bdf} not found in sysfs")] + GpuNotFound { bdf: String }, + + #[error("GPU {bdf} is not an NVIDIA device (vendor={vendor})")] + NotNvidia { bdf: String, vendor: String }, + + #[error("GPU {bdf} has no IOMMU group — is IOMMU enabled?")] + NoIommuGroup { bdf: String }, + + #[error("GPU {bdf} IOMMU group {group} has other non-vfio-pci devices: {peers:?}")] + IommuGroupConflict { + bdf: String, + group: u32, + peers: Vec, + }, + + #[error("failed to bind GPU {bdf} to vfio-pci: {reason}")] + BindFailed { bdf: String, reason: String }, + + #[error("failed to unbind GPU {bdf} from vfio-pci: {reason}")] + UnbindFailed { bdf: String, reason: String }, + + #[error("sysfs I/O error for {path}: {source}")] + SysfsIo { + path: String, + #[source] + source: std::io::Error, + }, + + #[error("invalid PCI BDF address: {bdf}")] + InvalidBdf { bdf: String }, +} + +// --------------------------------------------------------------------------- +// SysfsRoot +// --------------------------------------------------------------------------- + +/// Abstraction over sysfs paths, enabling test mocks via a temporary directory. +#[derive(Debug, Clone)] +pub struct SysfsRoot { + base: PathBuf, +} + +impl SysfsRoot { + /// Production root pointing at the real `/sys` filesystem. + pub fn system() -> Self { + Self { + base: PathBuf::from("/sys"), + } + } + + /// Custom root for testing. + pub fn new(base: impl Into) -> Self { + Self { base: base.into() } + } + + pub fn pci_devices_dir(&self) -> PathBuf { + self.base.join("bus/pci/devices") + } + + pub fn pci_device(&self, bdf: &str) -> PathBuf { + self.pci_devices_dir().join(bdf) + } + + pub fn drivers_probe(&self) -> PathBuf { + self.base.join("bus/pci/drivers_probe") + } + + pub fn vfio_pci_new_id(&self) -> PathBuf { + self.base.join("bus/pci/drivers/vfio-pci/new_id") + } + + pub fn vfio_pci_remove_id(&self) -> PathBuf { + self.base.join("bus/pci/drivers/vfio-pci/remove_id") + } + + pub fn iommu_group(&self, bdf: &str) -> Result { + let link = self.pci_device(bdf).join("iommu_group"); + let target = fs::read_link(&link).map_err(|_| VfioError::NoIommuGroup { + bdf: bdf.to_string(), + })?; + let group_str = + target + .file_name() + .and_then(|n| n.to_str()) + .ok_or_else(|| VfioError::NoIommuGroup { + bdf: bdf.to_string(), + })?; + group_str + .parse::() + .map_err(|_| VfioError::NoIommuGroup { + bdf: bdf.to_string(), + }) + } + + /// Enumerate all PCI BDFs in the given IOMMU group. + pub fn iommu_group_devices(&self, group_id: u32) -> Result, VfioError> { + let group_dir = self + .base + .join(format!("kernel/iommu_groups/{group_id}/devices")); + let entries = fs::read_dir(&group_dir).map_err(|source| VfioError::SysfsIo { + path: group_dir.display().to_string(), + source, + })?; + let mut devices = Vec::new(); + for entry in entries.filter_map(Result::ok) { + devices.push(entry.file_name().to_string_lossy().into_owned()); + } + devices.sort(); + Ok(devices) + } +} + +// --------------------------------------------------------------------------- +// GpuInfo +// --------------------------------------------------------------------------- + +/// Information about a discovered NVIDIA GPU eligible for VFIO passthrough. +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct GpuInfo { + pub bdf: String, + pub name: String, + pub vendor: String, + pub device: String, + pub iommu_group: u32, +} + +// --------------------------------------------------------------------------- +// GpuBindGuard +// --------------------------------------------------------------------------- + +/// RAII guard that restores a GPU to its host driver when dropped. +/// +/// Call [`disarm`](Self::disarm) to transfer ownership (e.g. the VM took over +/// the device successfully and we should not unbind it on cleanup). +pub struct GpuBindGuard { + bdf: String, + companion_bdfs: Vec, + sysfs: SysfsRoot, + disarmed: bool, + /// Cached "VVVV DDDD" string captured at bind time so that + /// deregistration from vfio-pci's match table succeeds even if the + /// device's sysfs entries have disappeared (e.g. physical removal). + vfio_id: Option, +} + +impl GpuBindGuard { + pub fn bdf(&self) -> &str { + &self.bdf + } + + /// Prevent the guard from restoring the GPU on drop. + pub fn disarm(mut self) { + self.disarmed = true; + } +} + +impl GpuBindGuard { + /// Deregister the cached vfio-pci match-table entry, then restore the + /// device to its host driver. + /// + /// Using the cached ID avoids re-reading vendor/device from sysfs, + /// which would fail if the GPU has been physically removed. + fn restore_with_cached_id(&self) { + if let Some(ref id_str) = self.vfio_id { + deregister_vfio_id_by_value(&self.sysfs, id_str); + } + + for peer in &self.companion_bdfs { + if let Err(err) = restore_gpu_to_host_driver(&self.sysfs, peer) { + tracing::error!(bdf = %peer, error = %err, "failed to restore companion device to host driver"); + } + } + + if let Err(err) = + restore_gpu_to_host_driver_ex(&self.sysfs, &self.bdf, self.vfio_id.is_some()) + { + tracing::error!(bdf = %self.bdf, error = %err, "failed to restore GPU to host driver"); + } + } +} + +impl Drop for GpuBindGuard { + fn drop(&mut self) { + if self.disarmed { + return; + } + if self.vfio_id.is_some() { + self.restore_with_cached_id(); + } else { + for peer in &self.companion_bdfs { + if let Err(err) = restore_gpu_to_host_driver(&self.sysfs, peer) { + tracing::error!(bdf = %peer, error = %err, "failed to restore companion device to host driver on drop"); + } + } + if let Err(err) = restore_gpu_to_host_driver(&self.sysfs, &self.bdf) { + tracing::error!(bdf = %self.bdf, error = %err, "failed to restore GPU to host driver on drop"); + } + } + } +} + +// --------------------------------------------------------------------------- +// GpuBindState (crash-recovery persistence) +// --------------------------------------------------------------------------- + +/// Persisted record of GPUs currently bound to vfio-pci, for crash recovery. +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct GpuBindState { + pub bindings: Vec, +} + +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct GpuBinding { + pub bdf: String, + pub sandbox_id: String, + pub bound_at_ms: i64, +} + +impl GpuBindState { + pub fn load(path: &Path) -> Result { + let data = fs::read_to_string(path)?; + serde_json::from_str(&data) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e)) + } + + pub fn save(&self, path: &Path) -> Result<(), std::io::Error> { + let data = serde_json::to_string_pretty(self) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?; + let tmp = path.with_extension("tmp"); + fs::write(&tmp, &data)?; + fs::rename(&tmp, path) + } +} + +// --------------------------------------------------------------------------- +// Validation helpers +// --------------------------------------------------------------------------- + +/// Validate a PCI BDF address (format `DDDD:BB:DD.F`). +pub fn validate_bdf(bdf: &str) -> Result<(), VfioError> { + let bytes = bdf.as_bytes(); + if bytes.len() != 12 { + return Err(VfioError::InvalidBdf { + bdf: bdf.to_string(), + }); + } + + // Expected layout: [hex×4]:[hex×2]:[hex×2].[hex×1] + // 0123 4 56 7 89 A B + let ok = is_hex(bytes[0]) + && is_hex(bytes[1]) + && is_hex(bytes[2]) + && is_hex(bytes[3]) + && bytes[4] == b':' + && is_hex(bytes[5]) + && is_hex(bytes[6]) + && bytes[7] == b':' + && is_hex(bytes[8]) + && is_hex(bytes[9]) + && bytes[10] == b'.' + && is_hex(bytes[11]); + + if ok { + Ok(()) + } else { + Err(VfioError::InvalidBdf { + bdf: bdf.to_string(), + }) + } +} + +fn is_hex(b: u8) -> bool { + b.is_ascii_hexdigit() +} + +/// Returns `true` if `data` contains only safe characters for sysfs values +/// (alphanumeric plus `:`, `.`, `-`, `_`). +pub fn validate_sysfs_data(data: &str) -> bool { + !data.is_empty() + && data + .chars() + .all(|c| c.is_alphanumeric() || matches!(c, ':' | '.' | '-' | '_')) +} + +// --------------------------------------------------------------------------- +// Sysfs helpers +// --------------------------------------------------------------------------- + +fn read_sysfs_trimmed(path: &Path) -> Result { + fs::read_to_string(path) + .map(|s| s.trim().to_string()) + .map_err(|source| VfioError::SysfsIo { + path: path.display().to_string(), + source, + }) +} + +fn write_sysfs(path: &Path, value: &str) -> Result<(), VfioError> { + fs::write(path, value).map_err(|source| VfioError::SysfsIo { + path: path.display().to_string(), + source, + }) +} + +fn current_driver_name(sysfs: &SysfsRoot, bdf: &str) -> Option { + let driver_link = sysfs.pci_device(bdf).join("driver"); + fs::read_link(&driver_link) + .ok() + .and_then(|p| p.file_name().map(|n| n.to_string_lossy().into_owned())) +} + +fn is_gpu_class(class_str: &str) -> bool { + if class_str == GPU_CLASS_DISPLAY_VGA { + return true; + } + // 3D controller: 0x0302xx + if let Some(hex) = class_str.strip_prefix("0x") + && let Ok(val) = u32::from_str_radix(hex, 16) + { + return (val >> 8) == GPU_CLASS_DISPLAY_3D; + } + false +} + +// --------------------------------------------------------------------------- +// Discovery +// --------------------------------------------------------------------------- + +/// Scan sysfs for NVIDIA GPUs eligible for VFIO passthrough. +pub fn probe_host_nvidia_vfio_readiness(sysfs: &SysfsRoot) -> Vec { + let devices_dir = sysfs.pci_devices_dir(); + let entries = match fs::read_dir(&devices_dir) { + Ok(e) => e, + Err(err) => { + tracing::warn!(path = %devices_dir.display(), %err, "cannot read PCI devices directory"); + return Vec::new(); + } + }; + + let mut gpus = Vec::new(); + + for entry in entries.filter_map(Result::ok) { + let bdf = entry.file_name().to_string_lossy().into_owned(); + let dev_dir = sysfs.pci_device(&bdf); + + let Ok(vendor) = read_sysfs_trimmed(&dev_dir.join("vendor")) else { + continue; + }; + if vendor != NVIDIA_VENDOR_ID { + continue; + } + + let Ok(class) = read_sysfs_trimmed(&dev_dir.join("class")) else { + continue; + }; + if !is_gpu_class(&class) { + continue; + } + + let device = read_sysfs_trimmed(&dev_dir.join("device")).unwrap_or_default(); + + let name = read_sysfs_trimmed(&dev_dir.join("label")) + .unwrap_or_else(|_| format!("NVIDIA {device}")); + + let Ok(iommu_group) = sysfs.iommu_group(&bdf) else { + continue; + }; + + gpus.push(GpuInfo { + bdf, + name, + vendor, + device, + iommu_group, + }); + } + + gpus +} + +// --------------------------------------------------------------------------- +// Bind / unbind +// --------------------------------------------------------------------------- + +/// Read vendor and device IDs from sysfs and format as `"VVVV DDDD"` (no `0x` prefix). +fn vfio_id_string(sysfs: &SysfsRoot, bdf: &str) -> Option { + let dev_dir = sysfs.pci_device(bdf); + let vendor = read_sysfs_trimmed(&dev_dir.join("vendor")).ok()?; + let device = read_sysfs_trimmed(&dev_dir.join("device")).ok()?; + let vendor_hex = vendor.strip_prefix("0x").unwrap_or(&vendor); + let device_hex = device.strip_prefix("0x").unwrap_or(&device); + Some(format!("{vendor_hex} {device_hex}")) +} + +/// Best-effort registration of a device's vendor:device ID with `vfio-pci`. +/// +/// Some kernel configurations require the ID to be pre-registered in +/// `/sys/bus/pci/drivers/vfio-pci/new_id` before `drivers_probe` will +/// bind the device, even when `driver_override` is set. Writing an +/// already-registered ID returns `EEXIST`, which we silently ignore. +fn register_vfio_new_id(sysfs: &SysfsRoot, bdf: &str) { + let Some(id_str) = vfio_id_string(sysfs, bdf) else { + return; + }; + + let should_write = { + let mut map = VFIO_ID_REFCOUNTS.lock().unwrap(); + let count = map.entry(id_str.clone()).or_insert(0); + *count += 1; + *count == 1 + }; + + if !should_write { + tracing::debug!( + bdf, id = %id_str, + "vfio-pci new_id already registered by another GPU, refcount incremented" + ); + return; + } + + let new_id_path = sysfs.vfio_pci_new_id(); + match write_sysfs(&new_id_path, &id_str) { + Ok(()) => { + tracing::debug!(bdf, id = %id_str, "registered vfio-pci new_id"); + } + Err(_) => { + tracing::debug!( + bdf, id = %id_str, + "vfio-pci new_id write skipped (already registered or driver not loaded)" + ); + } + } +} + +/// Best-effort deregistration of a device's vendor:device ID from `vfio-pci`. +/// +/// Reverses the effect of [`register_vfio_new_id`] by writing to +/// `/sys/bus/pci/drivers/vfio-pci/remove_id`. This prevents vfio-pci +/// from winning the probe race against the host driver when +/// `drivers_probe` runs during restore. +/// +/// `ENODEV` is silently ignored (the ID may never have been registered +/// or was already removed). +fn deregister_vfio_new_id(sysfs: &SysfsRoot, bdf: &str) { + let Some(id_str) = vfio_id_string(sysfs, bdf) else { + return; + }; + + let should_write = { + let mut map = VFIO_ID_REFCOUNTS.lock().unwrap(); + match map.get_mut(&id_str) { + Some(count) if *count > 1 => { + *count -= 1; + false + } + Some(_) => { + map.remove(&id_str); + true + } + None => true, + } + }; + + if !should_write { + tracing::debug!( + bdf, id = %id_str, + "vfio-pci remove_id skipped (other GPUs still using this ID)" + ); + return; + } + + let remove_id_path = sysfs.vfio_pci_remove_id(); + match write_sysfs(&remove_id_path, &id_str) { + Ok(()) => { + tracing::debug!(bdf, id = %id_str, "deregistered vfio-pci new_id"); + } + Err(_) => { + tracing::debug!( + bdf, id = %id_str, + "vfio-pci remove_id write skipped (not registered or already removed)" + ); + } + } +} + +/// Best-effort deregistration using a pre-captured ID string. +/// +/// Unlike [`deregister_vfio_new_id`], this does not read vendor/device +/// from sysfs at call time, making it reliable even when the device has +/// been physically removed or sysfs is otherwise inaccessible. +fn deregister_vfio_id_by_value(sysfs: &SysfsRoot, id_str: &str) { + let remove_id_path = sysfs.vfio_pci_remove_id(); + match write_sysfs(&remove_id_path, id_str) { + Ok(()) => { + tracing::debug!(id = %id_str, "deregistered vfio-pci new_id (by cached value)"); + } + Err(_) => { + tracing::debug!( + id = %id_str, + "vfio-pci remove_id write skipped (not registered or already removed)" + ); + } + } +} + +/// Bind a single PCI device to `vfio-pci`. Skips devices already bound. +fn bind_device_to_vfio(sysfs: &SysfsRoot, bdf: &str) -> Result { + if let Some(drv) = current_driver_name(sysfs, bdf) { + if drv == "vfio-pci" { + return Ok(false); + } + let unbind_path = sysfs.pci_device(bdf).join("driver/unbind"); + write_sysfs(&unbind_path, bdf).map_err(|e| VfioError::BindFailed { + bdf: bdf.to_string(), + reason: format!("unbind from {drv}: {e}"), + })?; + tracing::info!(bdf, driver = %drv, "unbound device from current driver"); + } + + register_vfio_new_id(sysfs, bdf); + + let override_path = sysfs.pci_device(bdf).join("driver_override"); + if let Err(e) = write_sysfs(&override_path, "vfio-pci") { + deregister_vfio_new_id(sysfs, bdf); + return Err(VfioError::BindFailed { + bdf: bdf.to_string(), + reason: format!("driver_override: {e}"), + }); + } + + if let Err(e) = write_sysfs(&sysfs.drivers_probe(), bdf) { + deregister_vfio_new_id(sysfs, bdf); + return Err(VfioError::BindFailed { + bdf: bdf.to_string(), + reason: format!("drivers_probe: {e}"), + }); + } + + if matches!(current_driver_name(sysfs, bdf).as_deref(), Some("vfio-pci")) { + return Ok(true); + } + + // The kernel processes drivers_probe asynchronously on some systems; + // poll briefly to let the driver attach before declaring failure. + for _ in 0..VFIO_BIND_MAX_POLL_ATTEMPTS { + std::thread::sleep(VFIO_BIND_POLL_INTERVAL); + if matches!(current_driver_name(sysfs, bdf).as_deref(), Some("vfio-pci")) { + tracing::debug!(bdf, "vfio-pci binding confirmed after polling"); + return Ok(true); + } + } + + deregister_vfio_new_id(sysfs, bdf); + Err(VfioError::BindFailed { + bdf: bdf.to_string(), + reason: format!( + "after drivers_probe with {}ms polling, driver is {:?} instead of vfio-pci", + u64::from(VFIO_BIND_MAX_POLL_ATTEMPTS) + * u64::try_from(VFIO_BIND_POLL_INTERVAL.as_millis()).unwrap_or(u64::MAX), + current_driver_name(sysfs, bdf) + .as_deref() + .unwrap_or("") + ), + }) +} + +/// Bind a GPU to `vfio-pci`, returning an RAII guard that restores it on drop. +/// +/// Also binds all companion devices in the same IOMMU group (e.g. the +/// HD Audio function on consumer GPUs). All bound companions are tracked +/// and restored when the guard is dropped. +pub fn prepare_gpu_for_passthrough( + sysfs: &SysfsRoot, + bdf: &str, +) -> Result { + validate_bdf(bdf)?; + + let dev_dir = sysfs.pci_device(bdf); + if !dev_dir.exists() { + return Err(VfioError::GpuNotFound { + bdf: bdf.to_string(), + }); + } + + let vendor = read_sysfs_trimmed(&dev_dir.join("vendor"))?; + if vendor != NVIDIA_VENDOR_ID { + return Err(VfioError::NotNvidia { + bdf: bdf.to_string(), + vendor, + }); + } + + let iommu_group = sysfs.iommu_group(bdf)?; + let group_devices = sysfs.iommu_group_devices(iommu_group)?; + let peers: Vec = group_devices.into_iter().filter(|d| d != bdf).collect(); + + let mut bound_companions = Vec::new(); + for peer in &peers { + if !sysfs.pci_device(peer).exists() { + continue; + } + match bind_device_to_vfio(sysfs, peer) { + Ok(was_bound) => { + if was_bound { + tracing::info!(bdf = %peer, iommu_group, "bound IOMMU group companion to vfio-pci"); + bound_companions.push(peer.clone()); + } + } + Err(err) => { + for already_bound in bound_companions.iter().rev() { + if let Err(restore_err) = restore_gpu_to_host_driver(sysfs, already_bound) { + tracing::error!(bdf = %already_bound, error = %restore_err, "failed to restore companion during rollback"); + } + } + return Err(VfioError::BindFailed { + bdf: peer.clone(), + reason: format!("IOMMU group {iommu_group} companion bind failed: {err}"), + }); + } + } + } + + match bind_device_to_vfio(sysfs, bdf) { + Ok(was_bound) => { + if was_bound { + tracing::info!(bdf, "GPU bound to vfio-pci"); + } else { + tracing::info!(bdf, "GPU already bound to vfio-pci"); + } + } + Err(err) => { + for companion in bound_companions.iter().rev() { + if let Err(restore_err) = restore_gpu_to_host_driver(sysfs, companion) { + tracing::error!(bdf = %companion, error = %restore_err, "failed to restore companion during rollback"); + } + } + return Err(err); + } + } + + let vfio_id = vfio_id_string(sysfs, bdf); + + Ok(GpuBindGuard { + bdf: bdf.to_string(), + companion_bdfs: bound_companions, + sysfs: sysfs.clone(), + disarmed: false, + vfio_id, + }) +} + +/// Restore a GPU from `vfio-pci` back to the host's default driver. +fn restore_gpu_to_host_driver(sysfs: &SysfsRoot, bdf: &str) -> Result<(), VfioError> { + restore_gpu_to_host_driver_ex(sysfs, bdf, false) +} + +/// Inner restore implementation. +/// +/// When `skip_deregister` is `true` the caller has already removed the +/// device's vendor:device ID from vfio-pci's match table (e.g. via a +/// cached value), so we skip the sysfs-based deregistration. +fn restore_gpu_to_host_driver_ex( + sysfs: &SysfsRoot, + bdf: &str, + skip_deregister: bool, +) -> Result<(), VfioError> { + let dev_dir = sysfs.pci_device(bdf); + + if !skip_deregister { + // Deregister the device ID from vfio-pci's match table before + // unbind+reprobe. Without this, drivers_probe re-binds to vfio-pci + // via the still-registered new_id entry. + deregister_vfio_new_id(sysfs, bdf); + } + + let unbind_path = dev_dir.join("driver/unbind"); + if unbind_path.exists() { + write_sysfs(&unbind_path, bdf).map_err(|e| VfioError::UnbindFailed { + bdf: bdf.to_string(), + reason: format!("unbind: {e}"), + })?; + } + + let override_path = dev_dir.join("driver_override"); + if override_path.exists() { + write_sysfs(&override_path, "\n").map_err(|e| VfioError::UnbindFailed { + bdf: bdf.to_string(), + reason: format!("clear driver_override: {e}"), + })?; + } + + let probe = sysfs.drivers_probe(); + if probe.exists() { + write_sysfs(&probe, bdf).map_err(|e| VfioError::UnbindFailed { + bdf: bdf.to_string(), + reason: format!("drivers_probe: {e}"), + })?; + } + + tracing::info!(bdf, "GPU restored to host driver"); + Ok(()) +} + +// --------------------------------------------------------------------------- +// Crash-recovery reconciliation +// --------------------------------------------------------------------------- + +/// Reconcile stale VFIO bindings left over from a previous crash. +/// +/// Loads persisted state, checks each GPU, and restores any that are still +/// bound to `vfio-pci`. Returns the list of BDFs that were restored. +/// Removes the state file only when all bindings are resolved; rewrites it +/// with the remaining entries when some restorations fail so they can be +/// retried on the next process start. +pub fn reconcile_stale_bindings(sysfs: &SysfsRoot, state_path: &Path) -> Vec { + let state = match GpuBindState::load(state_path) { + Ok(s) => s, + Err(err) => { + tracing::debug!(%err, path = %state_path.display(), "no stale GPU bind state to reconcile"); + return Vec::new(); + } + }; + + // Any in-memory refcounts are stale (from a previous process that + // crashed). Clear them so deregister writes through to sysfs. + if let Ok(mut map) = VFIO_ID_REFCOUNTS.lock() { + map.clear(); + } + + let mut restored = Vec::new(); + let mut failed_bindings = Vec::new(); + + for binding in &state.bindings { + match current_driver_name(sysfs, &binding.bdf) { + Some(ref drv) if drv == "vfio-pci" => { + tracing::warn!( + bdf = %binding.bdf, + sandbox_id = %binding.sandbox_id, + "stale VFIO binding detected, restoring GPU to host driver" + ); + if let Err(err) = restore_gpu_to_host_driver(sysfs, &binding.bdf) { + tracing::error!(bdf = %binding.bdf, %err, "failed to restore stale GPU binding"); + failed_bindings.push(binding.clone()); + continue; + } + restored.push(binding.bdf.clone()); + } + _ => { + let override_path = sysfs.pci_device(&binding.bdf).join("driver_override"); + if let Ok(val) = read_sysfs_trimmed(&override_path) + && val == "vfio-pci" + { + tracing::warn!( + bdf = %binding.bdf, + sandbox_id = %binding.sandbox_id, + "stale driver_override detected, clearing and re-probing" + ); + deregister_vfio_new_id(sysfs, &binding.bdf); + if let Err(err) = write_sysfs(&override_path, "\n") { + tracing::error!(bdf = %binding.bdf, %err, "failed to clear stale driver_override"); + failed_bindings.push(binding.clone()); + continue; + } + let probe = sysfs.drivers_probe(); + if let Err(err) = write_sysfs(&probe, &binding.bdf) { + tracing::error!(bdf = %binding.bdf, %err, "failed to re-probe after clearing driver_override"); + } + restored.push(binding.bdf.clone()); + } else { + tracing::debug!(bdf = %binding.bdf, "GPU no longer bound to vfio-pci, skipping"); + } + } + } + } + + if failed_bindings.is_empty() { + if let Err(err) = fs::remove_file(state_path) { + tracing::warn!(%err, path = %state_path.display(), "failed to remove stale bind state file"); + } + } else { + let remaining = GpuBindState { + bindings: failed_bindings, + }; + match serde_json::to_string_pretty(&remaining) { + Ok(json) => { + if let Err(err) = fs::write(state_path, json) { + tracing::error!(%err, path = %state_path.display(), "failed to persist remaining stale bindings"); + } else { + tracing::warn!( + count = remaining.bindings.len(), + "some GPU bindings could not be restored; state preserved for retry" + ); + } + } + Err(err) => { + tracing::error!(%err, "failed to serialize remaining stale bindings"); + } + } + } + + restored +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +#[cfg(test)] +mod tests { + use super::*; + use std::os::unix::fs::symlink; + use std::sync::{MutexGuard, PoisonError}; + use tempfile::TempDir; + + static VFIO_ID_REFCOUNT_TEST_LOCK: Mutex<()> = Mutex::new(()); + + fn vfio_id_refcount_test_guard() -> MutexGuard<'static, ()> { + VFIO_ID_REFCOUNT_TEST_LOCK + .lock() + .unwrap_or_else(PoisonError::into_inner) + } + + fn setup_mock_sysfs() -> (TempDir, SysfsRoot) { + let tmp = TempDir::new().unwrap(); + let sysfs = SysfsRoot::new(tmp.path()); + (tmp, sysfs) + } + + fn create_pci_device( + sysfs: &SysfsRoot, + tmp: &Path, + bdf: &str, + vendor: &str, + device: &str, + class: &str, + iommu_group: u32, + ) { + let dev = sysfs.pci_device(bdf); + fs::create_dir_all(&dev).unwrap(); + + fs::write(dev.join("vendor"), format!("{vendor}\n")).unwrap(); + fs::write(dev.join("device"), format!("{device}\n")).unwrap(); + fs::write(dev.join("class"), format!("{class}\n")).unwrap(); + + let group_dir = tmp.join(format!("kernel/iommu_groups/{iommu_group}")); + fs::create_dir_all(&group_dir).unwrap(); + symlink(&group_dir, dev.join("iommu_group")).unwrap(); + + let group_devices_dir = group_dir.join("devices"); + fs::create_dir_all(&group_devices_dir).unwrap(); + symlink(&dev, group_devices_dir.join(bdf)).unwrap(); + } + + /// Remove a specific vendor:device key from the global refcount map. + /// Used by tests to clean up their own entries without disturbing + /// parallel tests that hold refcounts for different device IDs. + fn clear_vfio_id_refcount(id: &str) { + VFIO_ID_REFCOUNTS.lock().unwrap().remove(id); + } + + // -- validate_bdf ------------------------------------------------------- + + #[test] + fn test_validate_bdf_valid() { + assert!(validate_bdf("0000:2d:00.0").is_ok()); + assert!(validate_bdf("0000:00:00.0").is_ok()); + assert!(validate_bdf("abcd:ef:01.a").is_ok()); + assert!(validate_bdf("ABCD:EF:01.A").is_ok()); + } + + #[test] + fn test_validate_bdf_invalid() { + assert!(validate_bdf("").is_err()); + assert!(validate_bdf("0000:2d:00").is_err()); // too short + assert!(validate_bdf("0000:2d:00.00").is_err()); // too long + assert!(validate_bdf("000g:2d:00.0").is_err()); // non-hex + assert!(validate_bdf("0000-2d-00.0").is_err()); // wrong separators + assert!(validate_bdf("0000:2d:00:0").is_err()); // colon instead of dot + } + + #[test] + fn test_validate_bdf_rejects_metacharacters() { + assert!(validate_bdf("$(rm -rf /)").is_err()); + assert!(validate_bdf("; echo pwned").is_err()); + assert!(validate_bdf("0000:2d;00.0").is_err()); + assert!(validate_bdf("0000:2d:0`.0").is_err()); + assert!(validate_bdf("../../../../").is_err()); + } + + // -- validate_sysfs_data ------------------------------------------------ + + #[test] + fn test_validate_sysfs_data() { + assert!(validate_sysfs_data("0x10de")); + assert!(validate_sysfs_data("vfio-pci")); + assert!(validate_sysfs_data("nvidia_gpu_0")); + assert!(validate_sysfs_data("0000:2d:00.0")); + + assert!(!validate_sysfs_data("")); + assert!(!validate_sysfs_data("$(echo)")); + assert!(!validate_sysfs_data("a b")); + assert!(!validate_sysfs_data("foo;bar")); + assert!(!validate_sysfs_data("a\nb")); + } + + // -- probe_host_nvidia_vfio_readiness ----------------------------------- + + #[test] + fn test_probe_discovers_nvidia_gpu() { + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 42, + ); + + let gpus = probe_host_nvidia_vfio_readiness(&sysfs); + assert_eq!(gpus.len(), 1); + assert_eq!(gpus[0].bdf, "0000:2d:00.0"); + assert_eq!(gpus[0].vendor, "0x10de"); + assert_eq!(gpus[0].device, "0x2684"); + assert_eq!(gpus[0].iommu_group, 42); + } + + #[test] + fn test_probe_skips_non_nvidia() { + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:01:00.0", + "0x8086", + "0x1234", + "0x030000", + 10, + ); + + let gpus = probe_host_nvidia_vfio_readiness(&sysfs); + assert!(gpus.is_empty()); + } + + #[test] + fn test_probe_skips_non_gpu_nvidia() { + let (tmp, sysfs) = setup_mock_sysfs(); + // Audio device class 0x040300 + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.1", + "0x10de", + "0x228b", + "0x040300", + 42, + ); + + let gpus = probe_host_nvidia_vfio_readiness(&sysfs); + assert!(gpus.is_empty()); + } + + // -- GpuBindState ------------------------------------------------------- + + #[test] + fn test_gpu_bind_state_roundtrip() { + let tmp = TempDir::new().unwrap(); + let path = tmp.path().join("gpu-state.json"); + + let state = GpuBindState { + bindings: vec![ + GpuBinding { + bdf: "0000:2d:00.0".to_string(), + sandbox_id: "sandbox-123".to_string(), + bound_at_ms: 1_700_000_000_000, + }, + GpuBinding { + bdf: "0000:3b:00.0".to_string(), + sandbox_id: "sandbox-456".to_string(), + bound_at_ms: 1_700_000_001_000, + }, + ], + }; + + state.save(&path).unwrap(); + let loaded = GpuBindState::load(&path).unwrap(); + + assert_eq!(loaded.bindings.len(), 2); + assert_eq!(loaded.bindings[0].bdf, "0000:2d:00.0"); + assert_eq!(loaded.bindings[0].sandbox_id, "sandbox-123"); + assert_eq!(loaded.bindings[1].bdf, "0000:3b:00.0"); + } + + // -- SysfsRoot ---------------------------------------------------------- + + #[test] + fn test_sysfs_root_paths() { + let sysfs = SysfsRoot::system(); + assert_eq!( + sysfs.pci_device("0000:2d:00.0"), + PathBuf::from("/sys/bus/pci/devices/0000:2d:00.0") + ); + assert_eq!( + sysfs.pci_devices_dir(), + PathBuf::from("/sys/bus/pci/devices") + ); + assert_eq!( + sysfs.drivers_probe(), + PathBuf::from("/sys/bus/pci/drivers_probe") + ); + assert_eq!( + sysfs.vfio_pci_new_id(), + PathBuf::from("/sys/bus/pci/drivers/vfio-pci/new_id") + ); + assert_eq!( + sysfs.vfio_pci_remove_id(), + PathBuf::from("/sys/bus/pci/drivers/vfio-pci/remove_id") + ); + + let custom = SysfsRoot::new("/tmp/test-sys"); + assert_eq!( + custom.pci_device("0000:01:00.0"), + PathBuf::from("/tmp/test-sys/bus/pci/devices/0000:01:00.0") + ); + } + + #[test] + fn test_sysfs_root_iommu_group() { + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 42, + ); + + assert_eq!(sysfs.iommu_group("0000:2d:00.0").unwrap(), 42); + assert!(sysfs.iommu_group("0000:ff:ff.f").is_err()); + } + + // -- is_gpu_class ------------------------------------------------------- + + #[test] + fn test_is_gpu_class() { + assert!(is_gpu_class("0x030000")); + assert!(is_gpu_class("0x030200")); + assert!(is_gpu_class("0x030201")); + assert!(!is_gpu_class("0x040300")); + assert!(!is_gpu_class("0x060000")); + assert!(!is_gpu_class("")); + } + + // -- iommu_group_devices ------------------------------------------------ + + #[test] + fn test_iommu_group_devices_lists_all_members() { + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 42, + ); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.1", + "0x10de", + "0x228b", + "0x040300", + 42, + ); + + let devices = sysfs.iommu_group_devices(42).unwrap(); + assert_eq!(devices.len(), 2); + assert!(devices.contains(&"0000:2d:00.0".to_string())); + assert!(devices.contains(&"0000:2d:00.1".to_string())); + } + + #[test] + fn test_iommu_group_devices_single_device() { + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 99, + ); + + let devices = sysfs.iommu_group_devices(99).unwrap(); + assert_eq!(devices.len(), 1); + assert_eq!(devices[0], "0000:2d:00.0"); + } + + // -- register_vfio_new_id ----------------------------------------------- + + #[test] + fn test_register_vfio_new_id_writes_vendor_device() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 26b3"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x26b3", + "0x030000", + 42, + ); + + let new_id_path = sysfs.vfio_pci_new_id(); + fs::create_dir_all(new_id_path.parent().unwrap()).unwrap(); + fs::write(&new_id_path, "").unwrap(); + + register_vfio_new_id(&sysfs, "0000:2d:00.0"); + + let written = fs::read_to_string(&new_id_path).unwrap(); + assert_eq!(written, "10de 26b3"); + } + + #[test] + fn test_register_vfio_new_id_ignores_missing_new_id_file() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 26b4"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x26b4", + "0x030000", + 42, + ); + + // Don't create the new_id file — should not panic or error + register_vfio_new_id(&sysfs, "0000:2d:00.0"); + } + + // -- deregister_vfio_new_id --------------------------------------------- + + #[test] + fn test_deregister_vfio_new_id_writes_vendor_device() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 26b5"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x26b5", + "0x030000", + 42, + ); + + let remove_id_path = sysfs.vfio_pci_remove_id(); + fs::create_dir_all(remove_id_path.parent().unwrap()).unwrap(); + fs::write(&remove_id_path, "").unwrap(); + + deregister_vfio_new_id(&sysfs, "0000:2d:00.0"); + + let written = fs::read_to_string(&remove_id_path).unwrap(); + assert_eq!(written, "10de 26b5"); + } + + #[test] + fn test_deregister_vfio_new_id_ignores_missing_remove_id_file() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 26b6"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x26b6", + "0x030000", + 42, + ); + + deregister_vfio_new_id(&sysfs, "0000:2d:00.0"); + } + + // -- refcount safety ---------------------------------------------------- + + #[test] + fn test_register_deregister_refcount() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 26b8"); + let (tmp, sysfs) = setup_mock_sysfs(); + + // Two GPUs with the same vendor:device ID (e.g., two A100s). + // Uses 0x26b8 — unique to this test to avoid parallel interference. + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x26b8", + "0x030000", + 42, + ); + create_pci_device( + &sysfs, + tmp.path(), + "0000:3b:00.0", + "0x10de", + "0x26b8", + "0x030200", + 43, + ); + + let new_id_path = sysfs.vfio_pci_new_id(); + fs::create_dir_all(new_id_path.parent().unwrap()).unwrap(); + fs::write(&new_id_path, "").unwrap(); + + let remove_id_path = sysfs.vfio_pci_remove_id(); + fs::write(&remove_id_path, "").unwrap(); + + // Register the same vendor:device for two different BDFs + register_vfio_new_id(&sysfs, "0000:2d:00.0"); + let written = fs::read_to_string(&new_id_path).unwrap(); + assert_eq!( + written, "10de 26b8", + "first register should write to new_id" + ); + + // Clear the file to detect whether the second register writes + fs::write(&new_id_path, "").unwrap(); + register_vfio_new_id(&sysfs, "0000:3b:00.0"); + let written = fs::read_to_string(&new_id_path).unwrap(); + assert_eq!( + written, "", + "second register should NOT write to new_id (refcount > 1)" + ); + + // Deregister once — should NOT write to remove_id (one GPU still using it) + deregister_vfio_new_id(&sysfs, "0000:2d:00.0"); + let written = fs::read_to_string(&remove_id_path).unwrap(); + assert_eq!( + written, "", + "first deregister should NOT write to remove_id" + ); + + // Deregister again — should write to remove_id (last user) + deregister_vfio_new_id(&sysfs, "0000:3b:00.0"); + let written = fs::read_to_string(&remove_id_path).unwrap(); + assert_eq!( + written, "10de 26b8", + "second deregister SHOULD write to remove_id" + ); + } + + // -- companion binding -------------------------------------------------- + + /// Helper to create a fake driver symlink for a mock PCI device. + fn set_mock_driver(sysfs: &SysfsRoot, bdf: &str, driver_name: &str) { + let driver_dir = sysfs.base.join(format!("bus/pci/drivers/{driver_name}")); + fs::create_dir_all(&driver_dir).unwrap(); + let dev_driver_link = sysfs.pci_device(bdf).join("driver"); + let _ = fs::remove_file(&dev_driver_link); + symlink(&driver_dir, &dev_driver_link).unwrap(); + } + + #[test] + fn test_prepare_gpu_skips_already_bound_companions() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 2684"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 42, + ); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.1", + "0x10de", + "0x228b", + "0x040300", + 42, + ); + + let probe = sysfs.drivers_probe(); + fs::create_dir_all(probe.parent().unwrap()).unwrap(); + fs::write(&probe, "").unwrap(); + + // Both devices already on vfio-pci + set_mock_driver(&sysfs, "0000:2d:00.0", "vfio-pci"); + set_mock_driver(&sysfs, "0000:2d:00.1", "vfio-pci"); + + let guard = prepare_gpu_for_passthrough(&sysfs, "0000:2d:00.0").unwrap(); + + // Both were already bound, no companions should be tracked for restore + assert!(guard.companion_bdfs.is_empty()); + assert_eq!(guard.bdf, "0000:2d:00.0"); + } + + #[test] + fn test_prepare_gpu_solo_iommu_group_no_companions() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 2684"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 99, + ); + + let probe = sysfs.drivers_probe(); + fs::create_dir_all(probe.parent().unwrap()).unwrap(); + fs::write(&probe, "").unwrap(); + + // GPU already on vfio-pci + set_mock_driver(&sysfs, "0000:2d:00.0", "vfio-pci"); + + let guard = prepare_gpu_for_passthrough(&sysfs, "0000:2d:00.0").unwrap(); + assert!(guard.companion_bdfs.is_empty()); + } + + #[test] + fn test_bind_device_to_vfio_already_bound() { + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 42, + ); + + let probe = sysfs.drivers_probe(); + fs::create_dir_all(probe.parent().unwrap()).unwrap(); + fs::write(&probe, "").unwrap(); + + set_mock_driver(&sysfs, "0000:2d:00.0", "vfio-pci"); + + let was_bound = bind_device_to_vfio(&sysfs, "0000:2d:00.0").unwrap(); + assert!(!was_bound, "should report false when already on vfio-pci"); + } + + #[test] + fn test_guard_drop_restores_companions() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 2684"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 42, + ); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.1", + "0x10de", + "0x228b", + "0x040300", + 42, + ); + + let probe = sysfs.drivers_probe(); + fs::create_dir_all(probe.parent().unwrap()).unwrap(); + fs::write(&probe, "").unwrap(); + + // Simulate bound state: driver link and driver_override both set + set_mock_driver(&sysfs, "0000:2d:00.0", "vfio-pci"); + set_mock_driver(&sysfs, "0000:2d:00.1", "vfio-pci"); + fs::write( + sysfs.pci_device("0000:2d:00.0").join("driver_override"), + "vfio-pci", + ) + .unwrap(); + fs::write( + sysfs.pci_device("0000:2d:00.1").join("driver_override"), + "vfio-pci", + ) + .unwrap(); + + { + let _guard = GpuBindGuard { + bdf: "0000:2d:00.0".to_string(), + companion_bdfs: vec!["0000:2d:00.1".to_string()], + sysfs: sysfs.clone(), + disarmed: false, + vfio_id: None, + }; + // guard drops here — should attempt restore on both devices + } + + // After drop, driver_override should be cleared (written with "\n") + let gpu_override = + fs::read_to_string(sysfs.pci_device("0000:2d:00.0").join("driver_override")).unwrap(); + assert_eq!( + gpu_override.trim(), + "", + "GPU driver_override should be cleared after drop" + ); + + let companion_override = + fs::read_to_string(sysfs.pci_device("0000:2d:00.1").join("driver_override")).unwrap(); + assert_eq!( + companion_override.trim(), + "", + "companion driver_override should be cleared after drop" + ); + } + + #[test] + fn test_guard_disarm_skips_restore() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 2684"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 42, + ); + + // Write a non-empty driver_override to verify it's NOT cleared + fs::write( + sysfs.pci_device("0000:2d:00.0").join("driver_override"), + "vfio-pci", + ) + .unwrap(); + + let guard = GpuBindGuard { + bdf: "0000:2d:00.0".to_string(), + companion_bdfs: vec![], + sysfs: sysfs.clone(), + disarmed: false, + vfio_id: None, + }; + guard.disarm(); + + // driver_override should still be vfio-pci (not cleared by disarmed guard) + let override_val = + fs::read_to_string(sysfs.pci_device("0000:2d:00.0").join("driver_override")).unwrap(); + assert_eq!(override_val, "vfio-pci"); + } + + // -- restore writes remove_id ------------------------------------------- + + #[test] + fn test_restore_gpu_deregisters_new_id_before_reprobe() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 26b7"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x26b7", + "0x030000", + 42, + ); + + let probe = sysfs.drivers_probe(); + fs::create_dir_all(probe.parent().unwrap()).unwrap(); + fs::write(&probe, "").unwrap(); + + let remove_id_path = sysfs.vfio_pci_remove_id(); + fs::create_dir_all(remove_id_path.parent().unwrap()).unwrap(); + fs::write(&remove_id_path, "").unwrap(); + + set_mock_driver(&sysfs, "0000:2d:00.0", "vfio-pci"); + fs::write( + sysfs.pci_device("0000:2d:00.0").join("driver_override"), + "vfio-pci", + ) + .unwrap(); + + restore_gpu_to_host_driver(&sysfs, "0000:2d:00.0").unwrap(); + + let written = fs::read_to_string(&remove_id_path).unwrap(); + assert_eq!( + written, "10de 26b7", + "remove_id should be written during restore" + ); + } + + // -- reconcile_stale_bindings ------------------------------------------- + + #[test] + fn test_reconcile_clears_stale_driver_override_when_not_on_vfio() { + let _refcount_guard = vfio_id_refcount_test_guard(); + clear_vfio_id_refcount("10de 2684"); + let (tmp, sysfs) = setup_mock_sysfs(); + create_pci_device( + &sysfs, + tmp.path(), + "0000:2d:00.0", + "0x10de", + "0x2684", + "0x030000", + 42, + ); + + let probe = sysfs.drivers_probe(); + fs::create_dir_all(probe.parent().unwrap()).unwrap(); + fs::write(&probe, "").unwrap(); + + set_mock_driver(&sysfs, "0000:2d:00.0", "nvidia"); + fs::write( + sysfs.pci_device("0000:2d:00.0").join("driver_override"), + "vfio-pci", + ) + .unwrap(); + + let state_path = tmp.path().join("gpu-state.json"); + let state = GpuBindState { + bindings: vec![GpuBinding { + bdf: "0000:2d:00.0".to_string(), + sandbox_id: "sandbox-orphan".to_string(), + bound_at_ms: 0, + }], + }; + state.save(&state_path).unwrap(); + + let restored = reconcile_stale_bindings(&sysfs, &state_path); + assert!(restored.contains(&"0000:2d:00.0".to_string())); + + let override_val = + fs::read_to_string(sysfs.pci_device("0000:2d:00.0").join("driver_override")).unwrap(); + assert_eq!( + override_val.trim(), + "", + "driver_override should be cleared even when device is not on vfio-pci" + ); + } +} diff --git a/crates/openshell-vm/README.md b/crates/openshell-vm/README.md index fcca20d5b9..266818741b 100644 --- a/crates/openshell-vm/README.md +++ b/crates/openshell-vm/README.md @@ -1,30 +1,32 @@ # openshell-vm -> Status: Experimental and work in progress (WIP). VM support is under active development and may change. +> Status: Legacy. This crate remains in the repository for later deprecation or +> removal, but it is excluded from normal workspace builds, CI, and release +> paths. Active VM sandbox work lives in `crates/openshell-driver-vm`. MicroVM runtime for OpenShell, powered by [libkrun](https://github.com/containers/libkrun). Boots a lightweight ARM64 Linux VM on macOS (Apple Hypervisor.framework) or Linux (KVM) running a single-node k3s cluster with the OpenShell control plane. -## Quick Start +## Current Path -```bash -# One-time setup: download pre-built runtime (~30s) -mise run vm:setup - -# Build and run the VM -mise run vm -``` +Use `mise run gateway:vm` for the supported per-sandbox VM driver workflow. The +standalone `openshell-vm` tasks and wrappers are intentionally not part of the +normal task surface. ## Prerequisites - **macOS (Apple Silicon)** or **Linux (aarch64 or x86_64 with KVM)** - Rust toolchain +- Guest-supervisor cross-compile toolchain (needed on macOS, and on Linux when host arch ≠ guest arch): + - Matching rustup target: `rustup target add aarch64-unknown-linux-gnu` (or `x86_64-unknown-linux-gnu` for an amd64 guest) + - `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `build-rootfs.sh` uses `cargo zigbuild` to cross-compile the in-VM `openshell-sandbox` supervisor binary. - [mise](https://mise.jdx.dev/) task runner - Docker (for rootfs builds) - `gh` CLI (for downloading pre-built runtime) ### macOS-Specific -The binary must be codesigned with the Hypervisor.framework entitlement. The `mise run vm` flow handles this automatically. To codesign manually: +The binary must be codesigned with the Hypervisor.framework entitlement. To +codesign manually: ```bash codesign --entitlements crates/openshell-vm/entitlements.plist --force -s - target/debug/openshell-vm @@ -34,7 +36,8 @@ codesign --entitlements crates/openshell-vm/entitlements.plist --force -s - targ ### Download Pre-Built Runtime (Default) -Downloads libkrun, libkrunfw, and gvproxy from the `vm-dev` GitHub Release: +Downloads libkrun, libkrunfw, and gvproxy from the `vm-runtime` GitHub Release for +the active VM driver runtime: ```bash mise run vm:setup @@ -52,25 +55,14 @@ On macOS this builds a custom libkrunfw (kernel firmware with bridge/netfilter s ## Build -Build the openshell-vm binary with embedded runtime: - -```bash -mise run vm:build -``` - -This compresses runtime artifacts, compiles the Rust binary with `include_bytes!()` embedding, codesigns it (macOS), and stages the sidecar runtime bundle. +There is no first-class `mise` build task for the standalone binary. This crate +is no longer part of normal CI or release builds. ## Rootfs -The rootfs is an Ubuntu filesystem containing k3s, pre-loaded container images, and the OpenShell binaries. Build it with: - -```bash -# Base rootfs (~200-300MB, cold starts in ~30-60s) -mise run vm:rootfs -- --base - -# Full rootfs (~2GB+, pre-initialized, boots in ~3-5s) -mise run vm:rootfs -``` +The legacy rootfs scripts are kept with this crate for historical reference. +They are not used by `openshell-driver-vm`, which derives each sandbox guest +rootfs from a container image at create time. ## Run @@ -78,11 +70,7 @@ mise run vm:rootfs Boots the full OpenShell gateway -- k3s + openshell-server + openshell-sandbox: -```bash -mise run vm -``` - -Or run the binary directly: +Run the binary directly after manually building and signing it: ```bash ./target/debug/openshell-vm @@ -118,7 +106,7 @@ Each instance gets its own extracted rootfs under `~/.local/share/openshell/open ## CLI Reference -``` +```text openshell-vm [OPTIONS] [COMMAND] Options: @@ -140,43 +128,15 @@ Subcommands: exec Execute a command inside a running VM ``` -## mise Tasks Reference - -| Task | Description | -|------|-------------| -| `vm` | Build and run the VM | -| `vm:build` | Build openshell-vm binary with embedded runtime | -| `vm:setup` | One-time setup: download (or build) the VM runtime | -| `vm:rootfs` | Build the VM rootfs tarball (`-- --base` for lightweight) | -| `vm:clean` | Remove all VM cached artifacts | -| `e2e:vm` | Boot VM and run smoke e2e tests | - -### Common Workflows - -```bash -# First time setup -mise run vm:setup # download pre-built runtime (~30s) -mise run vm # build + run +## Tasks -# Day-to-day iteration -mise run vm # incremental build + run - -# Need fresh rootfs -mise run vm:rootfs -- --base # rebuild base rootfs -mise run vm:build # rebuild binary with new rootfs - -# Something broken, start over -mise run vm:clean # wipe everything -mise run vm:setup # re-download runtime -mise run vm # full rebuild + run - -# Custom kernel work (rare) -FROM_SOURCE=1 mise run vm:setup -``` +Standalone `openshell-vm` tasks have been removed from the normal task surface. +The remaining VM tasks (`vm:setup`, `vm:supervisor`, `gateway:vm`, `e2e:vm`, +and `vm:smoke:orphan-cleanup`) support `openshell-driver-vm`. ## Architecture -``` +```text Host (macOS / Linux) openshell-vm binary |-- Embedded runtime (libkrun, libkrunfw, gvproxy, rootfs.tar.zst) diff --git a/crates/openshell-vm/build.rs b/crates/openshell-vm/build.rs index 33fab9a78c..6351be6e8a 100644 --- a/crates/openshell-vm/build.rs +++ b/crates/openshell-vm/build.rs @@ -12,7 +12,7 @@ //! Environment: //! `OPENSHELL_VM_RUNTIME_COMPRESSED_DIR` - Path to compressed artifacts -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use std::{env, fs}; fn main() { @@ -97,7 +97,7 @@ fn main() { e ) }); - let size = fs::metadata(&dst_path).map(|m| m.len()).unwrap_or(0); + let size = fs::metadata(&dst_path).map_or(0, |m| m.len()); println!("cargo:warning=Embedded {src_name}: {size} bytes"); } else { println!( @@ -116,7 +116,7 @@ fn main() { /// Generate stub (empty) resource files so the build can complete. /// The embedded module will fail at runtime if these stubs are used. -fn generate_stub_resources(out_dir: &PathBuf) { +fn generate_stub_resources(out_dir: &Path) { let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default(); let (libkrun_name, libkrunfw_name) = match target_os.as_str() { diff --git a/crates/openshell-vm/runtime/README.md b/crates/openshell-vm/runtime/README.md deleted file mode 100644 index f439811028..0000000000 --- a/crates/openshell-vm/runtime/README.md +++ /dev/null @@ -1,177 +0,0 @@ -# Custom libkrunfw Runtime - -> Status: Experimental and work in progress (WIP). VM support is under active development and may change. - -This directory contains the kernel config fragment for a custom `libkrunfw` runtime -that enables bridge CNI and netfilter support in the OpenShell gateway VM. - -## Why - -The stock `libkrunfw` (from Homebrew) ships a kernel without bridge, netfilter, -or conntrack support. This means the VM cannot: - -- Create `cni0` bridge interfaces (required by the bridge CNI plugin) -- Run kube-proxy (requires nftables) -- Route service VIP traffic (requires NAT/conntrack) - -The custom runtime builds libkrunfw with an additional kernel config fragment -that enables these networking and sandboxing features. - -## Directory Structure - -``` -runtime/ - kernel/ - openshell.kconfig # Kernel config fragment (networking + sandboxing) -``` - -## Build Pipeline - -Each platform builds its own kernel and runtime natively. - -``` -Linux ARM64: builds aarch64 kernel -> .so (parallel) -Linux AMD64: builds x86_64 kernel -> .so (parallel) -macOS ARM64: builds aarch64 kernel -> .dylib -``` - -### Build Scripts - -| Script | Platform | What it does | -|--------|----------|-------------| -| `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw + libkrun from source | -| `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Builds libkrunfw + libkrun from source | -| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages runtime tarball (libs + gvproxy + provenance) | - -### Quick Build (Linux) - -```bash -# Build both libkrunfw and libkrun from source -tasks/scripts/vm/build-libkrun.sh - -# Or build the full runtime from source via mise: -FROM_SOURCE=1 mise run vm:setup -``` - -### Quick Build (macOS) - -```bash -# Download pre-built runtime (recommended, ~30s): -mise run vm:setup - -# Or build from source: -FROM_SOURCE=1 mise run vm:setup -``` - -### Output - -Build artifacts are placed in `target/libkrun-build/`: - -``` -target/libkrun-build/ - libkrun.so / libkrun.dylib # The VMM library - libkrunfw.so* / libkrunfw.dylib # Kernel firmware library -``` - -## Networking - -The VM uses bridge CNI for pod networking with nftables-mode kube-proxy for -service VIP / ClusterIP support. The kernel config fragment enables both -iptables (for CNI bridge masquerade) and nftables (for kube-proxy). - -k3s is started with `--kube-proxy-arg=proxy-mode=nftables` because the -bundled iptables binaries in k3s have revision-negotiation issues with the -libkrun kernel's xt_MARK module. nftables mode uses the kernel's nf_tables -subsystem directly and avoids this entirely. - -## Runtime Provenance - -At VM boot, the openshell-vm binary logs provenance information about the loaded -runtime: - -``` -runtime: /path/to/openshell-vm.runtime - libkrunfw: libkrunfw.dylib - sha256: a1b2c3d4e5f6... - type: custom (OpenShell-built) - libkrunfw-commit: abc1234 - kernel-version: 6.6.30 - build-timestamp: 2026-03-23T10:00:00Z -``` - -For stock runtimes: -``` -runtime: /path/to/openshell-vm.runtime - libkrunfw: libkrunfw.dylib - sha256: f6e5d4c3b2a1... - type: stock (system/homebrew) -``` - -## Verification - -### Capability Check (inside VM) - -```bash -# Run inside the VM to verify kernel capabilities: -/srv/check-vm-capabilities.sh - -# JSON output for CI: -/srv/check-vm-capabilities.sh --json -``` - -### Rollback - -To revert to the stock runtime: - -```bash -# Unset the custom runtime source: -unset OPENSHELL_VM_RUNTIME_SOURCE_DIR - -# Re-download pre-built runtime and rebuild: -mise run vm:setup -mise run vm:build - -# Boot: -mise run vm -``` - -## Troubleshooting - -### "FailedCreatePodSandBox" bridge errors - -The kernel does not have bridge support. Verify: -```bash -# Inside VM: -ip link add test0 type bridge && echo "bridge OK" && ip link del test0 -``` - -If this fails, you are running the stock runtime. Build and use the custom one. - -### kube-proxy CrashLoopBackOff - -kube-proxy runs in nftables mode. If it crashes, verify nftables support: -```bash -# Inside VM: -nft list ruleset -``` - -If this fails, the kernel may lack `CONFIG_NF_TABLES`. Use the custom runtime. - -Common errors: -- `unknown option "--xor-mark"`: kube-proxy is running in iptables mode instead - of nftables. Verify `--kube-proxy-arg=proxy-mode=nftables` is in the k3s args. - -### Runtime mismatch after upgrade - -If libkrunfw is updated (e.g., via `brew upgrade`), the stock runtime may -change. Check provenance: -```bash -# Look for provenance info in VM boot output -grep "runtime:" ~/.local/share/openshell/openshell-vm/console.log -``` - -Re-build the custom runtime if needed: -```bash -FROM_SOURCE=1 mise run vm:setup -mise run vm:build -``` diff --git a/crates/openshell-vm/scripts/build-rootfs.sh b/crates/openshell-vm/scripts/build-rootfs.sh index d43046d4f4..bfafe8c85c 100755 --- a/crates/openshell-vm/scripts/build-rootfs.sh +++ b/crates/openshell-vm/scripts/build-rootfs.sh @@ -33,11 +33,20 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Source container engine abstraction (provides ce, ce_build, etc.) +_CE_SEARCH="${SCRIPT_DIR}/../../../tasks/scripts/container-engine.sh" +if [ -f "${_CE_SEARCH}" ]; then + source "${_CE_SEARCH}" +else + # Fallback: if run from a different working directory, try repo root. + source "$(cd "${SCRIPT_DIR}/../../.." && pwd)/tasks/scripts/container-engine.sh" +fi + # Source pinned dependency versions (digests, checksums, commit SHAs). -# Environment variables override pins — see pins.env for details. -PINS_FILE="${SCRIPT_DIR}/../pins.env" +# Environment variables override pins. +PINS_FILE="${SCRIPT_DIR}/../../openshell-driver-vm/runtime/pins.env" if [ -f "$PINS_FILE" ]; then - # shellcheck source=../pins.env + # shellcheck source=../../openshell-driver-vm/runtime/pins.env source "$PINS_FILE" fi @@ -119,6 +128,65 @@ verify_checksum() { fi } +ensure_build_nofile_limit() { + local desired="${OPENSHELL_VM_BUILD_NOFILE_LIMIT:-8192}" + local minimum=1024 + local current="" + local hard="" + local target="" + + [ "$(uname -s)" = "Darwin" ] || return 0 + command -v cargo-zigbuild >/dev/null 2>&1 || return 0 + + current="$(ulimit -n 2>/dev/null || echo "")" + case "${current}" in + ''|*[!0-9]*) + return 0 + ;; + esac + + if [ "${current}" -ge "${desired}" ]; then + return 0 + fi + + hard="$(ulimit -Hn 2>/dev/null || echo "")" + target="${desired}" + case "${hard}" in + ''|unlimited|infinity) + ;; + *[!0-9]*) + ;; + *) + if [ "${hard}" -lt "${target}" ]; then + target="${hard}" + fi + ;; + esac + + if [ "${target}" -gt "${current}" ] && ulimit -n "${target}" 2>/dev/null; then + echo "==> Raised open file limit for cargo-zigbuild: ${current} -> $(ulimit -n)" + fi + + current="$(ulimit -n 2>/dev/null || echo "${current}")" + case "${current}" in + ''|*[!0-9]*) + return 0 + ;; + esac + + if [ "${current}" -lt "${desired}" ]; then + echo "WARNING: Open file limit is ${current}; cargo-zigbuild is more reliable at ${desired}+ on macOS." + fi + + if [ "${current}" -lt "${minimum}" ]; then + echo "ERROR: Open file limit (${current}) is too low for cargo-zigbuild on macOS." + echo " Zig 0.14+ can fail with ProcessFdQuotaExceeded while linking large binaries." + echo " Run: ulimit -n ${desired}" + echo " Then re-run this script." + exit 1 + fi +} + if [ "$BASE_ONLY" = true ]; then echo "==> Building base openshell-vm rootfs" echo " Guest arch: ${GUEST_ARCH}" @@ -135,6 +203,10 @@ else fi echo "" +# cargo-zigbuild on macOS can exhaust the default per-process file descriptor +# limit while linking larger targets with Zig 0.14+. +ensure_build_nofile_limit + # ── Check for running VM ──────────────────────────────────────────────── # If an openshell-vm is using this rootfs via virtio-fs, wiping the rootfs # corrupts the VM's filesystem (e.g. /var disappears) causing cascading @@ -219,10 +291,10 @@ fi # ── Build base image with dependencies ───────────────────────────────── # Clean up any previous run -docker rm -f "${CONTAINER_NAME}" 2>/dev/null || true +ce rm -f "${CONTAINER_NAME}" 2>/dev/null || true echo "==> Building base image..." -docker build --platform "${DOCKER_PLATFORM}" -t "${BASE_IMAGE_TAG}" \ +ce build --platform "${DOCKER_PLATFORM}" -t "${BASE_IMAGE_TAG}" \ --build-arg "BASE_IMAGE=${VM_BASE_IMAGE}" -f - . <<'DOCKERFILE' ARG BASE_IMAGE FROM ${BASE_IMAGE} @@ -246,7 +318,7 @@ DOCKERFILE # Create a container and export the filesystem echo "==> Creating container..." -docker create --platform "${DOCKER_PLATFORM}" --name "${CONTAINER_NAME}" "${BASE_IMAGE_TAG}" /bin/true +ce create --platform "${DOCKER_PLATFORM}" --name "${CONTAINER_NAME}" "${BASE_IMAGE_TAG}" /bin/true echo "==> Exporting filesystem..." # Previous builds may leave overlayfs work/ dirs with permissions that @@ -256,9 +328,9 @@ if [ -d "${ROOTFS_DIR}" ]; then rm -rf "${ROOTFS_DIR}" fi mkdir -p "${ROOTFS_DIR}" -docker export "${CONTAINER_NAME}" | tar -C "${ROOTFS_DIR}" -xf - +ce export "${CONTAINER_NAME}" | tar -C "${ROOTFS_DIR}" -xf - -docker rm "${CONTAINER_NAME}" +ce rm "${CONTAINER_NAME}" # ── Inject k3s binary ──────────────────────────────────────────────── @@ -294,24 +366,36 @@ chmod +x "${ROOTFS_DIR}/srv/openshell-vm-exec-agent.py" # ── Build and inject openshell-sandbox supervisor binary ───────────── # The supervisor binary runs inside every sandbox pod. It is side-loaded # from the node filesystem via a read-only hostPath volume mount at -# /opt/openshell/bin. In the Docker-based gateway this is built in the -# Dockerfile.cluster supervisor-builder stage; here we cross-compile -# from the host using cargo-zigbuild. +# /opt/openshell/bin. Container images consume a prebuilt supervisor +# binary; here we cross-compile from the host using cargo-zigbuild. SUPERVISOR_TARGET="${RUST_TARGET}" SUPERVISOR_BIN="${PROJECT_ROOT}/target/${SUPERVISOR_TARGET}/release/openshell-sandbox" echo "==> Building openshell-sandbox supervisor binary (${SUPERVISOR_TARGET})..." -if command -v cargo-zigbuild >/dev/null 2>&1; then - cargo zigbuild --release -p openshell-sandbox --target "${SUPERVISOR_TARGET}" \ - --manifest-path "${PROJECT_ROOT}/Cargo.toml" 2>&1 | tail -5 +SUPERVISOR_BUILD_LOG="$(mktemp -t openshell-supervisor-build.XXXXXX.log)" +run_supervisor_build() { + if command -v cargo-zigbuild >/dev/null 2>&1; then + cargo zigbuild --release -p openshell-sandbox --target "${SUPERVISOR_TARGET}" \ + --manifest-path "${PROJECT_ROOT}/Cargo.toml" + else + # Fallback: use plain cargo build when cargo-zigbuild is not available. + # This works for native builds (e.g. building x86_64 on x86_64) but + # will fail for true cross-compilation without a cross toolchain. + echo " cargo-zigbuild not found, falling back to cargo build..." + cargo build --release -p openshell-sandbox --target "${SUPERVISOR_TARGET}" \ + --manifest-path "${PROJECT_ROOT}/Cargo.toml" + fi +} +if run_supervisor_build >"${SUPERVISOR_BUILD_LOG}" 2>&1; then + tail -5 "${SUPERVISOR_BUILD_LOG}" + rm -f "${SUPERVISOR_BUILD_LOG}" else - # Fallback: use plain cargo build when cargo-zigbuild is not available. - # This works for native builds (e.g. building x86_64 on x86_64) but - # will fail for true cross-compilation without a cross toolchain. - echo " cargo-zigbuild not found, falling back to cargo build..." - cargo build --release -p openshell-sandbox --target "${SUPERVISOR_TARGET}" \ - --manifest-path "${PROJECT_ROOT}/Cargo.toml" 2>&1 | tail -5 + status=$? + echo "ERROR: supervisor build failed. Full output:" >&2 + cat "${SUPERVISOR_BUILD_LOG}" >&2 + echo " (log saved at ${SUPERVISOR_BUILD_LOG})" >&2 + exit "${status}" fi if [ ! -f "${SUPERVISOR_BIN}" ]; then @@ -429,9 +513,9 @@ pull_and_save() { # Try to pull; if the registry is unavailable, fall back to the # local Docker image cache (image may exist from a previous pull). echo " pulling: ${image}..." - if ! docker pull --platform "${DOCKER_PLATFORM}" "${image}" --quiet 2>/dev/null; then - echo " pull failed, checking local Docker cache..." - if ! docker image inspect "${image}" >/dev/null 2>&1; then + if ! ce pull --platform "${DOCKER_PLATFORM}" "${image}" --quiet 2>/dev/null; then + echo " pull failed, checking local image cache..." + if ! ce image inspect "${image}" >/dev/null 2>&1; then echo "ERROR: image ${image} not available locally or from registry" exit 1 fi @@ -442,7 +526,7 @@ pull_and_save() { # Pipe through zstd for faster decompression and smaller tarballs. # k3s auto-imports .tar.zst files from the airgap images directory. # -T0 uses all CPU cores; -3 is a good speed/ratio tradeoff. - docker save "${image}" | zstd -T0 -3 -o "${output}" + ce save "${image}" | zstd -T0 -3 -o "${output}" # Cache for next rebuild. cp "${output}" "${cache}" } diff --git a/crates/openshell-vm/scripts/check-vm-capabilities.sh b/crates/openshell-vm/scripts/check-vm-capabilities.sh index 2e758f5e06..f88a1340cf 100755 --- a/crates/openshell-vm/scripts/check-vm-capabilities.sh +++ b/crates/openshell-vm/scripts/check-vm-capabilities.sh @@ -224,7 +224,7 @@ else echo "" echo "FAIL: $FAIL required capabilities missing." echo "The VM kernel needs to be rebuilt with the missing features." - echo "See: crates/openshell-vm/runtime/kernel/README.md" + echo "See: crates/openshell-driver-vm/runtime/README.md" exit 1 else echo "" diff --git a/crates/openshell-vm/scripts/openshell-vm-exec-agent.py b/crates/openshell-vm/scripts/openshell-vm-exec-agent.py index d7ffd81df4..f2b384cf9d 100644 --- a/crates/openshell-vm/scripts/openshell-vm-exec-agent.py +++ b/crates/openshell-vm/scripts/openshell-vm-exec-agent.py @@ -3,11 +3,15 @@ # SPDX-License-Identifier: Apache-2.0 import base64 +import fcntl import json import os +import pty import socket +import struct import subprocess import sys +import termios import threading @@ -42,6 +46,11 @@ def validate_env(env_items): return env +def set_winsize(fd, cols, rows): + winsize = struct.pack("HHHH", rows, cols, 0, 0) + fcntl.ioctl(fd, termios.TIOCSWINSZ, winsize) + + def stream_reader(pipe, frame_type, sock_file, lock): try: while True: @@ -79,6 +88,8 @@ def stdin_writer(proc, sock_file, sock, lock): proc.stdin.flush() elif kind == "stdin_close": break + elif kind == "resize": + pass else: send_frame( sock_file, @@ -96,14 +107,10 @@ def stdin_writer(proc, sock_file, sock, lock): pass -def handle_client(conn): - sock_file = conn.makefile("rwb", buffering=0) +def handle_client_pipe(conn, request, sock_file): + """Handle a client connection using pipes (non-TTY mode).""" lock = threading.Lock() try: - request = recv_line(sock_file) - if request is None: - return - argv = request.get("argv") or ["sh"] cwd = request.get("cwd") env = os.environ.copy() @@ -153,6 +160,148 @@ def handle_client(conn): conn.close() +def handle_client_tty(conn, request, sock_file): + """Handle a client connection with PTY allocation.""" + lock = threading.Lock() + master_fd = -1 + try: + argv = request.get("argv") or ["sh"] + cwd = request.get("cwd") + env = os.environ.copy() + env.update(validate_env(request.get("env") or [])) + env.setdefault("TERM", "xterm-256color") + + master_fd, slave_fd = pty.openpty() + + # Consume any resize frame sent right after the ExecRequest. + # The host sends it before starting the stdin pump, so it + # should arrive quickly. Use a short socket timeout. + conn.settimeout(0.5) + try: + pending = sock_file.readline() + if pending: + frame = json.loads(pending.decode("utf-8")) + if frame.get("type") == "resize": + set_winsize( + slave_fd, + frame.get("cols", 80), + frame.get("rows", 24), + ) + except (socket.timeout, ValueError, OSError): + pass + finally: + conn.settimeout(None) + + proc = subprocess.Popen( + argv, + cwd=cwd or "/", + env=env, + stdin=slave_fd, + stdout=slave_fd, + stderr=slave_fd, + preexec_fn=os.setsid, + ) + os.close(slave_fd) + + def pty_reader(): + try: + while True: + try: + chunk = os.read(master_fd, 8192) + except OSError: + break + if not chunk: + break + send_frame( + sock_file, + lock, + { + "type": "stdout", + "data": base64.b64encode(chunk).decode("ascii"), + }, + ) + except Exception: + pass + + def pty_stdin_writer(): + try: + while True: + frame = recv_line(sock_file) + if frame is None: + break + kind = frame.get("type") + if kind == "stdin": + payload = base64.b64decode(frame.get("data", "")) + try: + os.write(master_fd, payload) + except OSError: + break + elif kind == "resize": + try: + set_winsize( + master_fd, + frame.get("cols", 80), + frame.get("rows", 24), + ) + except OSError: + pass + elif kind == "stdin_close": + break + else: + send_frame( + sock_file, + lock, + {"type": "error", "message": f"unknown frame type: {kind}"}, + ) + break + except (BrokenPipeError, OSError): + pass + + reader_thread = threading.Thread(target=pty_reader, daemon=True) + stdin_thread = threading.Thread(target=pty_stdin_writer, daemon=True) + reader_thread.start() + stdin_thread.start() + + code = proc.wait() + reader_thread.join(timeout=2) + send_frame(sock_file, lock, {"type": "exit", "code": code}) + except Exception as exc: + try: + send_frame(sock_file, lock, {"type": "error", "message": str(exc)}) + except Exception: + pass + finally: + if master_fd >= 0: + try: + os.close(master_fd) + except OSError: + pass + try: + sock_file.close() + except Exception: + pass + conn.close() + + +def handle_client(conn): + sock_file = conn.makefile("rwb", buffering=0) + try: + request = recv_line(sock_file) + if request is None: + sock_file.close() + conn.close() + return + except Exception: + sock_file.close() + conn.close() + return + + if request.get("tty"): + handle_client_tty(conn, request, sock_file) + else: + handle_client_pipe(conn, request, sock_file) + + def main(): if not hasattr(socket, "AF_VSOCK"): print("AF_VSOCK is not available", file=sys.stderr) diff --git a/crates/openshell-vm/src/embedded.rs b/crates/openshell-vm/src/embedded.rs index 731f34b10e..537e7d7257 100644 --- a/crates/openshell-vm/src/embedded.rs +++ b/crates/openshell-vm/src/embedded.rs @@ -229,7 +229,8 @@ pub fn cleanup_old_rootfs() -> Result<(), VmError> { } let current_version_dir = base.join(VERSION); - cleanup_old_versions_in_base(&base, ¤t_version_dir) + cleanup_old_versions_in_base(&base, ¤t_version_dir); + Ok(()) } /// Check if the rootfs is embedded (non-empty). @@ -257,12 +258,13 @@ fn runtime_cache_key() -> String { let sample = &chunk[..chunk.len().min(64)]; let mut word: u64 = 0; for (j, &b) in sample.iter().enumerate() { - word ^= (b as u64) << ((j % 8) * 8); + word ^= u64::from(b) << ((j % 8) * 8); } // Mix in resource index so identical resources don't cancel out. - fp ^= word.rotate_left((i as u32) * 13 + 7); + let i_u32 = u32::try_from(i).unwrap_or(u32::MAX); + fp ^= word.rotate_left(i_u32 * 13 + 7); // Also mix in the total length so size changes are detected. - fp ^= (chunk.len() as u64).rotate_left((i as u32) * 17 + 3); + fp ^= (chunk.len() as u64).rotate_left(i_u32 * 17 + 3); } format!("{VERSION}-{fp:016x}") } @@ -286,17 +288,17 @@ fn rootfs_cache_base() -> Result { } fn cleanup_old_versions(current_dir: &Path) -> Result<(), VmError> { - cleanup_old_versions_in_base(&runtime_cache_base()?, current_dir) + cleanup_old_versions_in_base(&runtime_cache_base()?, current_dir); + Ok(()) } -fn cleanup_old_versions_in_base(base: &Path, current_dir: &Path) -> Result<(), VmError> { +fn cleanup_old_versions_in_base(base: &Path, current_dir: &Path) { if !base.exists() { - return Ok(()); + return; } - let entries = match fs::read_dir(base) { - Ok(e) => e, - Err(_) => return Ok(()), // Can't read, skip cleanup + let Ok(entries) = fs::read_dir(base) else { + return; // Can't read, skip cleanup }; for entry in entries.filter_map(Result::ok) { @@ -316,8 +318,6 @@ fn cleanup_old_versions_in_base(base: &Path, current_dir: &Path) -> Result<(), V } } } - - Ok(()) } fn extract_resource(compressed: &[u8], dest: &Path) -> Result<(), VmError> { @@ -417,7 +417,7 @@ fn validate_runtime_dir(dir: &Path) -> Result<(), VmError> { } // Check file is not empty (would indicate a stub) - let size = fs::metadata(path).map(|m| m.len()).unwrap_or(0); + let size = fs::metadata(path).map_or(0, |m| m.len()); if size == 0 { return Err(VmError::HostSetup(format!( "runtime file is empty (stub): {}", diff --git a/crates/openshell-vm/src/exec.rs b/crates/openshell-vm/src/exec.rs index 6195556e14..63771509e4 100644 --- a/crates/openshell-vm/src/exec.rs +++ b/crates/openshell-vm/src/exec.rs @@ -3,12 +3,14 @@ use std::fs::{self, File}; use std::io::{BufRead, BufReader, Read, Write}; +use std::os::fd::{AsFd, BorrowedFd}; use std::os::unix::net::UnixStream; use std::path::{Path, PathBuf}; use std::thread; use std::time::{SystemTime, UNIX_EPOCH}; use base64::Engine as _; +use nix::sys::termios::{self, SetArg, Termios}; use serde::{Deserialize, Serialize}; use crate::VmError; @@ -87,6 +89,7 @@ struct ExecRequest { enum ClientFrame { Stdin { data: String }, StdinClose, + Resize { cols: u16, rows: u16 }, } #[derive(Debug, Deserialize)] @@ -98,31 +101,72 @@ enum ServerFrame { Error { message: String }, } +struct RawModeGuard { + raw_fd: i32, + original: Termios, +} + +impl RawModeGuard { + fn enter() -> Result { + let stdin = std::io::stdin(); + let fd = stdin.as_fd(); + let original = + termios::tcgetattr(fd).map_err(|e| VmError::Exec(format!("tcgetattr: {e}")))?; + let mut raw = original.clone(); + termios::cfmakeraw(&mut raw); + termios::tcsetattr(fd, SetArg::TCSANOW, &raw) + .map_err(|e| VmError::Exec(format!("tcsetattr: {e}")))?; + Ok(Self { + raw_fd: std::os::unix::io::AsRawFd::as_raw_fd(&stdin), + original, + }) + } +} + +impl Drop for RawModeGuard { + fn drop(&mut self) { + let fd = unsafe { BorrowedFd::borrow_raw(self.raw_fd) }; + let _ = termios::tcsetattr(fd, SetArg::TCSANOW, &self.original); + } +} + +fn get_terminal_size() -> Option<(u16, u16)> { + let fd = std::os::unix::io::AsRawFd::as_raw_fd(&std::io::stdout()); + let mut ws: libc::winsize = unsafe { std::mem::zeroed() }; + let rc = unsafe { libc::ioctl(fd, libc::TIOCGWINSZ, &mut ws) }; + if rc == 0 && ws.ws_col > 0 && ws.ws_row > 0 { + Some((ws.ws_col, ws.ws_row)) + } else { + None + } +} + pub fn vm_exec_socket_path(rootfs: &Path) -> PathBuf { // Prefer XDG_RUNTIME_DIR (per-user, restricted permissions on Linux), // fall back to /tmp. Ownership/symlink validation happens in // secure_socket_base() when the gvproxy socket dir is created; here // we just compute the path. The parent directory is created (with // permission checks) at launch time via create_dir_all. - let base = if let Some(xdg) = std::env::var_os("XDG_RUNTIME_DIR") { - PathBuf::from(xdg) - } else { - let mut base = PathBuf::from("/tmp"); - if !base.is_dir() { - base = std::env::temp_dir(); - } - base - }; + let base = std::env::var_os("XDG_RUNTIME_DIR").map_or_else( + || { + let mut base = PathBuf::from("/tmp"); + if !base.is_dir() { + base = std::env::temp_dir(); + } + base + }, + PathBuf::from, + ); let dir = base.join("ovm-exec"); let id = hash_path_id(rootfs); dir.join(format!("{id}.sock")) } fn hash_path_id(path: &Path) -> String { - let mut hash: u64 = 0xcbf29ce484222325; + let mut hash: u64 = 0xcbf2_9ce4_8422_2325; for byte in path.to_string_lossy().as_bytes() { hash ^= u64::from(*byte); - hash = hash.wrapping_mul(0x100000001b3); + hash = hash.wrapping_mul(0x0100_0000_01b3); } format!("{:012x}", hash & 0x0000_ffff_ffff_ffff) } @@ -302,15 +346,15 @@ pub fn reset_runtime_state(rootfs: &Path, gateway_name: &str) -> Result<(), VmEr /// This function is a no-op if `state.db` does not exist (e.g. first boot or /// after a full `--reset`). pub fn recover_corrupt_kine_db(rootfs: &Path) -> Result<(), VmError> { + // The SQLite file format begins with a 16-byte magic string. + // Reference: https://www.sqlite.org/fileformat.html#the_database_header + const SQLITE_MAGIC: &[u8] = b"SQLite format 3\x00"; + let db_path = rootfs.join("var/lib/rancher/k3s/server/db/state.db"); if !db_path.exists() { return Ok(()); // Nothing to check — first boot or post-reset. } - // The SQLite file format begins with a 16-byte magic string. - // Reference: https://www.sqlite.org/fileformat.html#the_database_header - const SQLITE_MAGIC: &[u8] = b"SQLite format 3\x00"; - // Read only the first 100 bytes (the minimum valid SQLite header size) // instead of loading the entire database into memory. let has_invalid_header = match File::open(&db_path).and_then(|mut f| { @@ -495,9 +539,19 @@ pub fn exec_running_vm(options: VmExecOptions) -> Result { }; send_json_line(&mut writer, &request)?; + let tty = options.tty; + let _raw_guard = if tty { + if let Some((cols, rows)) = get_terminal_size() { + send_json_line(&mut writer, &ClientFrame::Resize { cols, rows })?; + } + Some(RawModeGuard::enter()?) + } else { + None + }; + let stdin_writer = writer; thread::spawn(move || { - let _ = pump_stdin(stdin_writer); + let _ = pump_stdin(stdin_writer, tty); }); let mut reader = BufReader::new(&mut stream); @@ -724,10 +778,11 @@ fn send_json_line(writer: &mut UnixStream, value: &T) -> Result<() .map_err(|e| VmError::Exec(format!("write VM exec request: {e}"))) } -fn pump_stdin(mut writer: UnixStream) -> Result<(), VmError> { +fn pump_stdin(mut writer: UnixStream, tty: bool) -> Result<(), VmError> { let stdin = std::io::stdin(); let mut stdin = stdin.lock(); let mut buf = [0u8; 8192]; + let mut last_size: Option<(u16, u16)> = None; loop { let read = stdin @@ -736,6 +791,21 @@ fn pump_stdin(mut writer: UnixStream) -> Result<(), VmError> { if read == 0 { break; } + + if tty + && let Some(size) = get_terminal_size() + && last_size != Some(size) + { + last_size = Some(size); + let _ = send_json_line( + &mut writer, + &ClientFrame::Resize { + cols: size.0, + rows: size.1, + }, + ); + } + let frame = ClientFrame::Stdin { data: base64::engine::general_purpose::STANDARD.encode(&buf[..read]), }; @@ -765,3 +835,342 @@ fn now_ms() -> Result { .map_err(|e| VmError::RuntimeState(format!("read system clock: {e}")))?; Ok(duration.as_millis()) } + +#[cfg(test)] +mod tests { + use super::*; + + // ── ExecRequest serialization ──────────────────────────────────── + + #[test] + fn exec_request_serializes_with_tty() { + let req = ExecRequest { + argv: vec!["sh".into()], + env: vec!["TERM=xterm".into()], + cwd: None, + tty: true, + }; + let json: serde_json::Value = serde_json::to_value(&req).unwrap(); + assert_eq!(json["argv"], serde_json::json!(["sh"])); + assert_eq!(json["tty"], true); + assert_eq!(json["cwd"], serde_json::Value::Null); + } + + #[test] + fn exec_request_serializes_without_tty() { + let req = ExecRequest { + argv: vec!["echo".into(), "hello".into()], + env: vec![], + cwd: Some("/tmp".into()), + tty: false, + }; + let json: serde_json::Value = serde_json::to_value(&req).unwrap(); + assert_eq!(json["tty"], false); + assert_eq!(json["cwd"], "/tmp"); + } + + // ── ClientFrame serialization ──────────────────────────────────── + + #[test] + fn client_frame_stdin_serializes() { + let frame = ClientFrame::Stdin { + data: "aGVsbG8=".into(), + }; + let json: serde_json::Value = serde_json::to_value(&frame).unwrap(); + assert_eq!(json["type"], "stdin"); + assert_eq!(json["data"], "aGVsbG8="); + } + + #[test] + fn client_frame_stdin_close_serializes() { + let frame = ClientFrame::StdinClose; + let json: serde_json::Value = serde_json::to_value(&frame).unwrap(); + assert_eq!(json["type"], "stdin_close"); + } + + #[test] + fn client_frame_resize_serializes() { + let frame = ClientFrame::Resize { + cols: 120, + rows: 40, + }; + let json: serde_json::Value = serde_json::to_value(&frame).unwrap(); + assert_eq!(json["type"], "resize"); + assert_eq!(json["cols"], 120); + assert_eq!(json["rows"], 40); + } + + // ── ServerFrame deserialization ─────────────────────────────────── + + #[test] + fn server_frame_stdout_deserializes() { + let json = r#"{"type":"stdout","data":"aGVsbG8="}"#; + let frame: ServerFrame = serde_json::from_str(json).unwrap(); + assert!(matches!(frame, ServerFrame::Stdout { data } if data == "aGVsbG8=")); + } + + #[test] + fn server_frame_stderr_deserializes() { + let json = r#"{"type":"stderr","data":"ZXJy"}"#; + let frame: ServerFrame = serde_json::from_str(json).unwrap(); + assert!(matches!(frame, ServerFrame::Stderr { data } if data == "ZXJy")); + } + + #[test] + fn server_frame_exit_deserializes() { + let json = r#"{"type":"exit","code":42}"#; + let frame: ServerFrame = serde_json::from_str(json).unwrap(); + assert!(matches!(frame, ServerFrame::Exit { code: 42 })); + } + + #[test] + fn server_frame_error_deserializes() { + let json = r#"{"type":"error","message":"boom"}"#; + let frame: ServerFrame = serde_json::from_str(json).unwrap(); + assert!(matches!(frame, ServerFrame::Error { message } if message == "boom")); + } + + #[test] + fn server_frame_unknown_type_fails() { + let json = r#"{"type":"unknown","data":"x"}"#; + assert!(serde_json::from_str::(json).is_err()); + } + + // ── ClientFrame ↔ ServerFrame round-trip compatibility ─────────── + // Verify that what the Rust host serializes can be parsed by the + // Python agent (same JSON shape), and vice versa. + + #[test] + fn resize_frame_has_expected_json_shape() { + let frame = ClientFrame::Resize { cols: 80, rows: 24 }; + let s = serde_json::to_string(&frame).unwrap(); + let v: serde_json::Value = serde_json::from_str(&s).unwrap(); + assert_eq!(v["type"].as_str().unwrap(), "resize"); + assert!(v["cols"].is_u64()); + assert!(v["rows"].is_u64()); + } + + // ── validate_env_vars ──────────────────────────────────────────── + + #[test] + fn validate_env_vars_accepts_valid() { + let items = vec![ + "HOME=/root".to_string(), + "PATH=/usr/bin".to_string(), + "_UNDERSCORE=1".to_string(), + "A1B2=val".to_string(), + ]; + assert!(validate_env_vars(&items).is_ok()); + } + + #[test] + fn validate_env_vars_rejects_missing_equals() { + let items = vec!["NOEQUALS".to_string()]; + assert!(validate_env_vars(&items).is_err()); + } + + #[test] + fn validate_env_vars_rejects_empty_key() { + let items = vec!["=value".to_string()]; + assert!(validate_env_vars(&items).is_err()); + } + + #[test] + fn validate_env_vars_rejects_leading_digit() { + let items = vec!["1BAD=val".to_string()]; + assert!(validate_env_vars(&items).is_err()); + } + + #[test] + fn validate_env_vars_rejects_special_chars() { + let items = vec!["BAD-KEY=val".to_string()]; + assert!(validate_env_vars(&items).is_err()); + } + + // ── decode_payload ─────────────────────────────────────────────── + + #[test] + fn decode_payload_valid_base64() { + let decoded = decode_payload("aGVsbG8=").unwrap(); + assert_eq!(decoded, b"hello"); + } + + #[test] + fn decode_payload_empty() { + let decoded = decode_payload("").unwrap(); + assert!(decoded.is_empty()); + } + + #[test] + fn decode_payload_invalid_base64() { + assert!(decode_payload("!!!not-base64!!!").is_err()); + } + + // ── Resize frame edge cases ────────────────────────────────────── + + #[test] + fn resize_frame_max_dimensions() { + let frame = ClientFrame::Resize { + cols: u16::MAX, + rows: u16::MAX, + }; + let json: serde_json::Value = serde_json::to_value(&frame).unwrap(); + assert_eq!(json["cols"], u64::from(u16::MAX)); + assert_eq!(json["rows"], u64::from(u16::MAX)); + } + + #[test] + fn resize_frame_minimum_dimensions() { + let frame = ClientFrame::Resize { cols: 1, rows: 1 }; + let json: serde_json::Value = serde_json::to_value(&frame).unwrap(); + assert_eq!(json["cols"], 1); + assert_eq!(json["rows"], 1); + } + + // ── Wire format: newline-delimited JSON ────────────────────────── + // The protocol sends one JSON object per line. Verify that + // serialized frames produce valid single-line JSON that the + // Python agent can split on '\n' and json.loads(). + + #[test] + fn client_frames_serialize_to_single_line_json() { + let frames: Vec = vec![ + ClientFrame::Stdin { + data: "dGVzdA==".into(), + }, + ClientFrame::StdinClose, + ClientFrame::Resize { cols: 80, rows: 24 }, + ]; + for frame in &frames { + let s = serde_json::to_string(frame).unwrap(); + assert!(!s.contains('\n'), "frame should be single-line: {s}"); + let _: serde_json::Value = serde_json::from_str(&s).unwrap(); + } + } + + #[test] + fn exec_request_serializes_to_single_line_json() { + let req = ExecRequest { + argv: vec!["bash".into(), "-c".into(), "echo 'hello world'".into()], + env: vec!["HOME=/root".into(), "TERM=xterm-256color".into()], + cwd: Some("/home/user".into()), + tty: true, + }; + let s = serde_json::to_string(&req).unwrap(); + assert!(!s.contains('\n')); + let _: serde_json::Value = serde_json::from_str(&s).unwrap(); + } + + // ── Stdin data encode → decode round-trip ──────────────────────── + // Mirrors the flow: host encodes payload as base64 in a Stdin + // frame, guest decodes with decode_payload(). + + #[test] + fn stdin_payload_round_trip() { + let original = b"echo hello\n"; + let encoded = base64::engine::general_purpose::STANDARD.encode(original); + let frame = ClientFrame::Stdin { data: encoded }; + let json = serde_json::to_string(&frame).unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&json).unwrap(); + let decoded = decode_payload(parsed["data"].as_str().unwrap()).unwrap(); + assert_eq!(decoded, original); + } + + #[test] + fn stdin_payload_round_trip_binary() { + let original: Vec = (0..=255).collect(); + let encoded = base64::engine::general_purpose::STANDARD.encode(&original); + let decoded = decode_payload(&encoded).unwrap(); + assert_eq!(decoded, original); + } + + // ── Python agent compatibility ─────────────────────────────────── + // The Python agent parses frames with json.loads() and dispatches + // on frame["type"]. These tests verify the exact field names and + // values match what the Python code expects. + + #[test] + fn exec_request_tty_field_matches_python_dispatch() { + // Python: request.get("tty") — must be a JSON boolean + let req = ExecRequest { + argv: vec!["sh".into()], + env: vec![], + cwd: None, + tty: true, + }; + let v: serde_json::Value = serde_json::to_value(&req).unwrap(); + assert!(v["tty"].is_boolean()); + assert!(v["tty"].as_bool().unwrap()); + + let req_no_tty = ExecRequest { + argv: vec!["echo".into()], + env: vec![], + cwd: None, + tty: false, + }; + let v: serde_json::Value = serde_json::to_value(&req_no_tty).unwrap(); + assert!(!v["tty"].as_bool().unwrap()); + } + + #[test] + fn resize_type_tag_is_snake_case() { + // Python: kind == "resize" — must be lowercase snake_case + let frame = ClientFrame::Resize { cols: 80, rows: 24 }; + let v: serde_json::Value = serde_json::to_value(&frame).unwrap(); + assert_eq!(v["type"].as_str().unwrap(), "resize"); + } + + #[test] + fn stdin_close_type_tag_is_snake_case() { + // Python: kind == "stdin_close" + let frame = ClientFrame::StdinClose; + let v: serde_json::Value = serde_json::to_value(&frame).unwrap(); + assert_eq!(v["type"].as_str().unwrap(), "stdin_close"); + } + + #[test] + fn resize_fields_are_integers_not_strings() { + // Python: frame.get("cols", 80) — expects int, not string + let frame = ClientFrame::Resize { + cols: 200, + rows: 50, + }; + let v: serde_json::Value = serde_json::to_value(&frame).unwrap(); + assert!(v["cols"].is_u64()); + assert!(v["rows"].is_u64()); + } + + // ── ServerFrame: Python agent output ───────────────────────────── + // These mirror the exact JSON the Python agent produces with + // json.dumps(frame, separators=(",", ":")) + + #[test] + fn server_frame_parses_compact_json() { + // Python uses separators=(",", ":") — no spaces + let compact = r#"{"type":"stdout","data":"aGk="}"#; + let frame: ServerFrame = serde_json::from_str(compact).unwrap(); + assert!(matches!(frame, ServerFrame::Stdout { data } if data == "aGk=")); + } + + #[test] + fn server_frame_exit_code_zero() { + let json = r#"{"type":"exit","code":0}"#; + let frame: ServerFrame = serde_json::from_str(json).unwrap(); + assert!(matches!(frame, ServerFrame::Exit { code: 0 })); + } + + #[test] + fn server_frame_exit_code_negative() { + let json = r#"{"type":"exit","code":-1}"#; + let frame: ServerFrame = serde_json::from_str(json).unwrap(); + assert!(matches!(frame, ServerFrame::Exit { code: -1 })); + } + + #[test] + fn server_frame_tolerates_extra_fields() { + // Future-proofing: agent may add fields we don't know about + let json = r#"{"type":"exit","code":0,"extra":"ignored"}"#; + let frame: ServerFrame = serde_json::from_str(json).unwrap(); + assert!(matches!(frame, ServerFrame::Exit { code: 0 })); + } +} diff --git a/crates/openshell-vm/src/ffi.rs b/crates/openshell-vm/src/ffi.rs index 7500b1c973..66213c6247 100644 --- a/crates/openshell-vm/src/ffi.rs +++ b/crates/openshell-vm/src/ffi.rs @@ -93,6 +93,7 @@ type KrunAddNetUnixstream = unsafe extern "C" fn( flags: u32, ) -> i32; +#[allow(clippy::struct_field_names)] // FFI struct mirrors libkrun's symbol naming pub struct LibKrun { pub krun_init_log: KrunInitLog, pub krun_create_ctx: KrunCreateCtx, @@ -207,7 +208,10 @@ fn preload_runtime_support_libraries(runtime_dir: &Path) -> Result, .is_some_and(|name| { #[cfg(target_os = "macos")] { - name.starts_with("libkrunfw") && name.ends_with(".dylib") + name.starts_with("libkrunfw") + && Path::new(name) + .extension() + .is_some_and(|ext| ext.eq_ignore_ascii_case("dylib")) } #[cfg(not(target_os = "macos"))] { diff --git a/crates/openshell-vm/src/health.rs b/crates/openshell-vm/src/health.rs index 096a35d1f9..c24015bf19 100644 --- a/crates/openshell-vm/src/health.rs +++ b/crates/openshell-vm/src/health.rs @@ -14,8 +14,11 @@ use std::path::PathBuf; use std::time::Duration; use tonic::transport::{Certificate, ClientTlsConfig, Endpoint, Identity}; +/// CA certificate, client certificate, and client key bytes for mTLS. +type MtlsMaterials = (Vec, Vec, Vec); + /// Load mTLS materials from the gateway's cert directory. -fn load_mtls_materials(gateway_name: &str) -> Result<(Vec, Vec, Vec), String> { +fn load_mtls_materials(gateway_name: &str) -> Result { let home = std::env::var("HOME").map_err(|_| "HOME not set")?; let mtls_dir = PathBuf::from(home) .join(".config/openshell/gateways") diff --git a/crates/openshell-vm/src/lib.rs b/crates/openshell-vm/src/lib.rs index 2b78a7669c..2f8280b9cc 100644 --- a/crates/openshell-vm/src/lib.rs +++ b/crates/openshell-vm/src/lib.rs @@ -611,7 +611,7 @@ impl VmContext { Ok(Self { krun, - ctx_id: ctx_id as u32, + ctx_id: ctx_id.cast_unsigned(), }) } @@ -736,8 +736,8 @@ impl VmContext { } fn set_port_map(&self, port_map: &[String]) -> Result<(), VmError> { - let port_strs: Vec<&str> = port_map.iter().map(String::as_str).collect(); - let (_port_owners, port_ptrs) = c_string_array(&port_strs)?; + let port_refs: Vec<&str> = port_map.iter().map(String::as_str).collect(); + let (_port_owners, port_ptrs) = c_string_array(&port_refs)?; unsafe { check( (self.krun.krun_set_port_map)(self.ctx_id, port_ptrs.as_ptr()), @@ -773,10 +773,10 @@ impl VmContext { fn set_exec(&self, exec_path: &str, args: &[String], env: &[String]) -> Result<(), VmError> { let exec_c = CString::new(exec_path)?; - let argv_strs: Vec<&str> = args.iter().map(String::as_str).collect(); - let (_argv_owners, argv_ptrs) = c_string_array(&argv_strs)?; - let env_strs: Vec<&str> = env.iter().map(String::as_str).collect(); - let (_env_owners, env_ptrs) = c_string_array(&env_strs)?; + let argv_refs: Vec<&str> = args.iter().map(String::as_str).collect(); + let (_argv_owners, argv_ptrs) = c_string_array(&argv_refs)?; + let env_refs: Vec<&str> = env.iter().map(String::as_str).collect(); + let (_env_owners, env_ptrs) = c_string_array(&env_refs)?; unsafe { check( @@ -942,7 +942,7 @@ fn kill_stale_gvproxy_by_port(port: u16) { for line in pids.lines() { if let Ok(pid) = line.trim().parse::() { - let pid_i32 = pid as libc::pid_t; + let pid_i32 = pid.cast_signed(); if is_process_named(pid_i32, "gvproxy") { kill_gvproxy_pid(pid); } @@ -951,7 +951,7 @@ fn kill_stale_gvproxy_by_port(port: u16) { } fn kill_gvproxy_pid(gvproxy_pid: u32) { - let pid_i32 = gvproxy_pid as libc::pid_t; + let pid_i32 = gvproxy_pid.cast_signed(); let is_alive = unsafe { libc::kill(pid_i32, 0) } == 0; if is_alive { // Verify the process is actually gvproxy before killing. @@ -998,9 +998,7 @@ fn is_process_named(pid: libc::pid_t, expected: &str) -> bool { #[cfg(target_os = "linux")] fn is_process_named(pid: libc::pid_t, expected: &str) -> bool { let comm_path = format!("/proc/{pid}/comm"); - std::fs::read_to_string(comm_path) - .map(|name| name.trim().contains(expected)) - .unwrap_or(false) + std::fs::read_to_string(comm_path).is_ok_and(|name| name.trim().contains(expected)) } #[cfg(not(any(target_os = "macos", target_os = "linux")))] @@ -1090,10 +1088,10 @@ fn state_disk_sync_mode() -> u32 { } fn hash_path_id(path: &Path) -> String { - let mut hash: u64 = 0xcbf29ce484222325; + let mut hash: u64 = 0xcbf2_9ce4_8422_2325; for byte in path.to_string_lossy().as_bytes() { hash ^= u64::from(*byte); - hash = hash.wrapping_mul(0x100000001b3); + hash = hash.wrapping_mul(0x0100_0000_01b3); } format!("{:012x}", hash & 0x0000_ffff_ffff_ffff) } @@ -1104,15 +1102,16 @@ fn hash_path_id(path: &Path) -> String { /// falls back to `/tmp`. After `create_dir_all`, validates the directory /// is not a symlink and is owned by the current user. fn secure_socket_base(subdir: &str) -> Result { - let base = if let Some(xdg) = std::env::var_os("XDG_RUNTIME_DIR") { - PathBuf::from(xdg) - } else { - let mut base = PathBuf::from("/tmp"); - if !base.is_dir() { - base = std::env::temp_dir(); - } - base - }; + let base = std::env::var_os("XDG_RUNTIME_DIR").map_or_else( + || { + let mut base = PathBuf::from("/tmp"); + if !base.is_dir() { + base = std::env::temp_dir(); + } + base + }, + PathBuf::from, + ); let dir = base.join(subdir); // If the path exists, verify it is not a symlink before using it. @@ -1431,29 +1430,23 @@ pub fn launch(config: &VmConfig) -> Result { // network stack to misroute or drop packets. let mac: [u8; 6] = [0x5a, 0x94, 0xef, 0xe4, 0x0c, 0xee]; - // COMPAT_NET_FEATURES from libkrun.h - const NET_FEATURE_CSUM: u32 = 1 << 0; - const NET_FEATURE_GUEST_CSUM: u32 = 1 << 1; - const NET_FEATURE_GUEST_TSO4: u32 = 1 << 7; - const NET_FEATURE_GUEST_UFO: u32 = 1 << 10; - const NET_FEATURE_HOST_TSO4: u32 = 1 << 11; - const NET_FEATURE_HOST_UFO: u32 = 1 << 14; - const COMPAT_NET_FEATURES: u32 = NET_FEATURE_CSUM - | NET_FEATURE_GUEST_CSUM - | NET_FEATURE_GUEST_TSO4 - | NET_FEATURE_GUEST_UFO - | NET_FEATURE_HOST_TSO4 - | NET_FEATURE_HOST_UFO; + // COMPAT_NET_FEATURES from libkrun.h: + // NET_FEATURE_CSUM (1 << 0) | NET_FEATURE_GUEST_CSUM (1 << 1) + // | NET_FEATURE_GUEST_TSO4 (1 << 7) | NET_FEATURE_GUEST_UFO (1 << 10) + // | NET_FEATURE_HOST_TSO4 (1 << 11) | NET_FEATURE_HOST_UFO (1 << 14). + let compat_net_features: u32 = + (1 << 0) | (1 << 1) | (1 << 7) | (1 << 10) | (1 << 11) | (1 << 14); // On Linux use unixstream (SOCK_STREAM) to connect to gvproxy's // QEMU listener. On macOS use unixgram (SOCK_DGRAM) with the vfkit // magic byte for the vfkit listener. #[cfg(target_os = "linux")] - vm.add_net_unixstream(&net_sock, &mac, COMPAT_NET_FEATURES)?; + vm.add_net_unixstream(&net_sock, &mac, compat_net_features)?; #[cfg(target_os = "macos")] { - const NET_FLAG_VFKIT: u32 = 1 << 0; - vm.add_net_unixgram(&net_sock, &mac, COMPAT_NET_FEATURES, NET_FLAG_VFKIT)?; + // NET_FLAG_VFKIT = 1 << 0 + let net_flag_vfkit: u32 = 1 << 0; + vm.add_net_unixgram(&net_sock, &mac, compat_net_features, net_flag_vfkit)?; } eprintln!( @@ -1733,13 +1726,9 @@ fn bootstrap_gateway(rootfs: &Path, gateway_name: &str, gateway_port: u16) -> Re let metadata = openshell_bootstrap::GatewayMetadata { name: gateway_name.to_string(), gateway_endpoint: format!("https://127.0.0.1:{gateway_port}"), - is_remote: false, gateway_port, - remote_host: None, - resolved_host: None, - auth_mode: None, - edge_team_domain: None, - edge_auth_url: None, + client_lifecycle_managed: Some(true), + ..Default::default() }; let exec_socket = vm_exec_socket_path(rootfs); @@ -1761,15 +1750,11 @@ fn bootstrap_gateway(rootfs: &Path, gateway_name: &str, gateway_port: u16) -> Re // drift check and the host already has valid certs. If the agent // isn't reachable we skip silently rather than blocking boot for // 30s. - match fetch_pki_over_exec(&exec_socket, std::time::Duration::from_secs(5)) { - Ok(bundle) => { - if let Err(e) = sync_host_certs_if_stale(gateway_name, &bundle) { - eprintln!("Warning: cert sync check failed: {e}"); - } - } - Err(_) => { - // Expected on warm boot — exec agent not ready yet. - } + // Expected on warm boot — exec agent not ready yet. + if let Ok(bundle) = fetch_pki_over_exec(&exec_socket, std::time::Duration::from_secs(5)) + && let Err(e) = sync_host_certs_if_stale(gateway_name, &bundle) + { + eprintln!("Warning: cert sync check failed: {e}"); } eprintln!( diff --git a/crates/openshell-vm/src/main.rs b/crates/openshell-vm/src/main.rs index bb9d854b12..b2dce993ef 100644 --- a/crates/openshell-vm/src/main.rs +++ b/crates/openshell-vm/src/main.rs @@ -127,31 +127,31 @@ fn main() { // is available and the variable is not already configured. #[cfg(target_os = "macos")] { - if std::env::var_os("__OPENSHELL_VM_REEXEC").is_none() { - if let Ok(runtime_dir) = openshell_vm::configured_runtime_dir() { - let needs_reexec = std::env::var_os("DYLD_LIBRARY_PATH").map_or(true, |v| { - !v.to_string_lossy() - .contains(runtime_dir.to_str().unwrap_or("")) - }); - if needs_reexec { - let mut dyld_paths = vec![runtime_dir]; - if let Some(existing) = std::env::var_os("DYLD_LIBRARY_PATH") { - dyld_paths.extend(std::env::split_paths(&existing)); - } - let joined = std::env::join_paths(&dyld_paths).expect("join DYLD_LIBRARY_PATH"); - let exe = std::env::current_exe().expect("current_exe"); - let args: Vec = std::env::args().skip(1).collect(); - let err = std::process::Command::new(exe) - .args(&args) - .env("DYLD_LIBRARY_PATH", &joined) - .env("__OPENSHELL_VM_REEXEC", "1") - .status(); - match err { - Ok(status) => std::process::exit(status.code().unwrap_or(1)), - Err(e) => { - eprintln!("Error: failed to re-exec with DYLD_LIBRARY_PATH: {e}"); - std::process::exit(1); - } + if std::env::var_os("__OPENSHELL_VM_REEXEC").is_none() + && let Ok(runtime_dir) = openshell_vm::configured_runtime_dir() + { + let needs_reexec = std::env::var_os("DYLD_LIBRARY_PATH").is_none_or(|v| { + !v.to_string_lossy() + .contains(runtime_dir.to_str().unwrap_or("")) + }); + if needs_reexec { + let mut dyld_paths = vec![runtime_dir]; + if let Some(existing) = std::env::var_os("DYLD_LIBRARY_PATH") { + dyld_paths.extend(std::env::split_paths(&existing)); + } + let joined = std::env::join_paths(&dyld_paths).expect("join DYLD_LIBRARY_PATH"); + let exe = std::env::current_exe().expect("current_exe"); + let args: Vec = std::env::args().skip(1).collect(); + let err = std::process::Command::new(exe) + .args(&args) + .env("DYLD_LIBRARY_PATH", &joined) + .env("__OPENSHELL_VM_REEXEC", "1") + .status(); + match err { + Ok(status) => std::process::exit(status.code().unwrap_or(1)), + Err(e) => { + eprintln!("Error: failed to re-exec with DYLD_LIBRARY_PATH: {e}"); + std::process::exit(1); } } } diff --git a/crates/openshell-vm/tests/gateway_integration.rs b/crates/openshell-vm/tests/gateway_integration.rs index 7ababb42f1..fabfd74c82 100644 --- a/crates/openshell-vm/tests/gateway_integration.rs +++ b/crates/openshell-vm/tests/gateway_integration.rs @@ -62,7 +62,7 @@ fn assert_runtime_bundle_staged() { /// Boot the full `OpenShell` gateway and verify the gRPC service becomes /// reachable on port 30051. #[test] -#[ignore] // requires libkrun + rootfs +#[ignore = "requires libkrun + rootfs"] fn gateway_boots_and_service_becomes_reachable() { codesign_if_needed(); assert_runtime_bundle_staged(); @@ -87,7 +87,7 @@ fn gateway_boots_and_service_becomes_reachable() { } // Tear down regardless of result. - let _ = unsafe { libc::kill(child.id() as i32, libc::SIGTERM) }; + let _ = unsafe { libc::kill(child.id().cast_signed(), libc::SIGTERM) }; let _ = child.wait(); assert!( @@ -99,7 +99,7 @@ fn gateway_boots_and_service_becomes_reachable() { /// Run a trivial command inside the VM via `--exec` and verify it exits /// successfully, proving the VM boots and can execute guest processes. #[test] -#[ignore] // requires libkrun + rootfs +#[ignore = "requires libkrun + rootfs"] fn gateway_exec_runs_guest_command() { codesign_if_needed(); assert_runtime_bundle_staged(); @@ -119,7 +119,7 @@ fn gateway_exec_runs_guest_command() { /// Boot the VM, then use `openshell-vm exec` against the running instance. #[test] -#[ignore] // requires libkrun + rootfs +#[ignore = "requires libkrun + rootfs"] fn gateway_exec_attaches_to_running_vm() { codesign_if_needed(); assert_runtime_bundle_staged(); @@ -143,7 +143,7 @@ fn gateway_exec_attaches_to_running_vm() { .output() .expect("failed to run openshell-vm exec"); - let _ = unsafe { libc::kill(child.id() as i32, libc::SIGTERM) }; + let _ = unsafe { libc::kill(child.id().cast_signed(), libc::SIGTERM) }; let _ = child.wait(); assert!( diff --git a/deploy/deb/control.in b/deploy/deb/control.in new file mode 100644 index 0000000000..9f77f8775d --- /dev/null +++ b/deploy/deb/control.in @@ -0,0 +1,16 @@ +Package: openshell +Version: @VERSION@ +Architecture: @ARCH@ +Maintainer: NVIDIA OpenShell Maintainers +Section: utils +Priority: optional +Pre-Depends: init-system-helpers (>= 1.54~) +Homepage: https://github.com/NVIDIA/OpenShell +Description: Safe, sandboxed runtimes for autonomous AI agents + OpenShell provides host-side command-line and gateway components for + launching and managing policy-enforced AI agent sandboxes. The package + installs the openshell CLI, openshell-gateway daemon, and the VM compute + driver helper, plus a per-user systemd unit for running the gateway. + . + The systemd unit is user-scope (under /usr/lib/systemd/user/). To start + it, run `systemctl --user enable --now openshell-gateway`. diff --git a/deploy/deb/openshell-gateway.service b/deploy/deb/openshell-gateway.service new file mode 100644 index 0000000000..4705b547d2 --- /dev/null +++ b/deploy/deb/openshell-gateway.service @@ -0,0 +1,28 @@ +[Unit] +Description=OpenShell Gateway +Documentation=https://github.com/NVIDIA/OpenShell +After=default.target + +[Service] +Type=simple +StateDirectory=openshell/gateway +# %S resolves to $XDG_STATE_HOME for user services. +Environment=OPENSHELL_BIND_ADDRESS=127.0.0.1 +Environment=OPENSHELL_SERVER_PORT=17670 +Environment=OPENSHELL_DISABLE_TLS=true +Environment=OPENSHELL_DISABLE_GATEWAY_AUTH=true +Environment=OPENSHELL_DB_URL=sqlite:%S/openshell/gateway/openshell.db +Environment=OPENSHELL_DRIVERS=vm +Environment=OPENSHELL_GRPC_ENDPOINT=http://127.0.0.1:17670 +Environment=OPENSHELL_SSH_GATEWAY_HOST=127.0.0.1 +Environment=OPENSHELL_SSH_GATEWAY_PORT=17670 +Environment=OPENSHELL_VM_DRIVER_STATE_DIR=%S/openshell/vm-driver +EnvironmentFile=-%h/.config/openshell/gateway.env +ExecStart=/usr/bin/openshell-gateway +Restart=on-failure +RestartSec=5s +PrivateTmp=true +UMask=0077 + +[Install] +WantedBy=default.target diff --git a/deploy/deb/postinst.sh b/deploy/deb/postinst.sh new file mode 100755 index 0000000000..d9784214f9 --- /dev/null +++ b/deploy/deb/postinst.sh @@ -0,0 +1,34 @@ +#!/bin/sh +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# postinst for openshell. +# +# The packaged systemd unit is user-scope (installed under +# /usr/lib/systemd/user/) so dpkg cannot enable or start it on the user's +# behalf. Each user opts in by running: +# systemctl --user daemon-reload +# systemctl --user enable --now openshell-gateway +# +# Tell any running per-user systemd manager to re-scan its unit search +# path so already-logged-in users see the new unit without restarting +# their session. +set -e + +case "$1" in +configure | abort-upgrade | abort-deconfigure | abort-remove) + if [ -d /run/systemd/system ] && command -v systemctl >/dev/null 2>&1; then + systemctl daemon-reload >/dev/null 2>&1 || true + # --global only refreshes per-user managers' generators on next + # session; existing managers pick up new unit files without it. + systemctl --global daemon-reload >/dev/null 2>&1 || true + fi + ;; + +*) + echo "postinst called with unknown argument \`$1'" >&2 + exit 1 + ;; +esac + +exit 0 diff --git a/deploy/deb/postrm.sh b/deploy/deb/postrm.sh new file mode 100755 index 0000000000..7e1893abeb --- /dev/null +++ b/deploy/deb/postrm.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# postrm for openshell. +set -e + +if [ -d /run/systemd/system ] && command -v systemctl >/dev/null 2>&1; then + systemctl daemon-reload >/dev/null 2>&1 || true + systemctl --global daemon-reload >/dev/null 2>&1 || true +fi + +exit 0 diff --git a/deploy/deb/prerm.sh b/deploy/deb/prerm.sh new file mode 100755 index 0000000000..e9099eac64 --- /dev/null +++ b/deploy/deb/prerm.sh @@ -0,0 +1,21 @@ +#!/bin/sh +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# prerm for openshell. +# +# Per-user systemd units cannot be stopped from a system-scope dpkg hook. +# Users who want to stop their gateway before removing the package should +# run `systemctl --user stop openshell-gateway` themselves. +set -e + +case "$1" in +remove | upgrade | deconfigure | failed-upgrade) ;; + +*) + echo "prerm called with unknown argument \`$1'" >&2 + exit 1 + ;; +esac + +exit 0 diff --git a/deploy/docker/Dockerfile.ci b/deploy/docker/Dockerfile.ci index b09c93e119..3c669a96f7 100644 --- a/deploy/docker/Dockerfile.ci +++ b/deploy/docker/Dockerfile.ci @@ -8,8 +8,9 @@ FROM nvcr.io/nvidia/base/ubuntu:noble-20251013 -ARG DOCKER_VERSION=29.3.1 -ARG BUILDX_VERSION=v0.32.1 +ARG DOCKER_VERSION=29.4.1 +ARG BUILDX_VERSION=v0.33.0 +ARG NPM_VERSION=11.13.0 ARG TARGETARCH ENV DEBIAN_FRONTEND=noninteractive @@ -55,7 +56,7 @@ RUN case "$TARGETARCH" in \ && chmod +x /usr/local/lib/docker/cli-plugins/docker-buildx # Install GitHub CLI used by install.sh and CI jobs -ARG GH_VERSION=2.74.1 +ARG GH_VERSION=2.91.0 RUN case "$TARGETARCH" in \ amd64) gh_arch=amd64 ;; \ arm64) gh_arch=arm64 ;; \ @@ -64,18 +65,21 @@ RUN case "$TARGETARCH" in \ && curl -fsSL "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_${gh_arch}.tar.gz" \ | tar xz --strip-components=2 -C /usr/local/bin "gh_${GH_VERSION}_linux_${gh_arch}/bin/gh" -# Install mise -ARG MISE_VERSION=v2026.3.13 +# Install mise (NOTE: keep this version in sync with mise.toml) +ARG MISE_VERSION=v2026.4.25 RUN curl https://mise.run | MISE_VERSION=$MISE_VERSION sh # Copy mise.toml and task includes, then install all tools via mise COPY mise.toml /opt/mise/mise.toml +COPY mise.lock /opt/mise/mise.lock COPY tasks/ /opt/mise/tasks/ WORKDIR /opt/mise RUN --mount=type=secret,id=MISE_GITHUB_TOKEN \ export MISE_GITHUB_TOKEN="$(cat /run/secrets/MISE_GITHUB_TOKEN 2>/dev/null || true)" && \ mise trust /opt/mise/mise.toml && \ - env -u RUSTC_WRAPPER mise install && \ + env -u RUSTC_WRAPPER mise install --locked && \ + mise reshim && \ + npm install -g "npm@${NPM_VERSION}" && \ mise reshim && \ (/root/.cargo/bin/rustup component remove rust-docs || true) && \ rm -rf /root/.rustup/toolchains/*/share/doc /root/.rustup/toolchains/*/share/man diff --git a/deploy/docker/Dockerfile.cli-macos b/deploy/docker/Dockerfile.cli-macos index f9370691cb..d07d5a9eeb 100644 --- a/deploy/docker/Dockerfile.cli-macos +++ b/deploy/docker/Dockerfile.cli-macos @@ -37,7 +37,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ pkg-config \ && rm -rf /var/lib/apt/lists/* -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.88.0 +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.95.0 # aws-lc-sys probes with --target=arm64-apple-macosx and clang then looks for # arm64-apple-macosx-ld. Provide a linker alias to osxcross ld64. @@ -102,6 +102,7 @@ RUN --mount=type=cache,id=cargo-registry-cli-macos,sharing=locked,target=/root/. # Stage 2: real build # --------------------------------------------------------------------------- COPY crates/ crates/ +COPY providers/ providers/ # Touch source files to ensure they're rebuilt (not the cached dummy). RUN touch crates/openshell-cli/src/main.rs \ diff --git a/deploy/docker/Dockerfile.vm-macos b/deploy/docker/Dockerfile.driver-vm-macos similarity index 55% rename from deploy/docker/Dockerfile.vm-macos rename to deploy/docker/Dockerfile.driver-vm-macos index c033e43e84..f667653d0c 100644 --- a/deploy/docker/Dockerfile.vm-macos +++ b/deploy/docker/Dockerfile.driver-vm-macos @@ -3,17 +3,17 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Cross-compile the openshell-vm binary for macOS aarch64 (Apple Silicon) -# using the osxcross toolchain. +# Cross-compile the openshell-driver-vm binary for macOS aarch64 (Apple +# Silicon) using the osxcross toolchain. # -# The openshell-vm binary loads libkrun/libkrunfw at runtime via dlopen, so it -# does NOT need Hypervisor.framework headers at build time. Pre-compressed -# runtime artifacts (libkrun, libkrunfw, gvproxy, rootfs) are injected via +# openshell-driver-vm loads libkrun/libkrunfw at runtime via dlopen, so it +# does NOT need Hypervisor.framework headers at build time. Pre-compressed +# runtime artifacts (libkrun, libkrunfw, gvproxy, bundled supervisor) are injected via # the vm-runtime-compressed build context and embedded into the binary via # include_bytes!(). # # Usage: -# docker buildx build -f deploy/docker/Dockerfile.vm-macos \ +# docker buildx build -f deploy/docker/Dockerfile.driver-vm-macos \ # --build-arg OPENSHELL_CARGO_VERSION=0.6.0 \ # --build-context vm-runtime-compressed=/path/to/compressed-dir \ # --output type=local,dest=out/ . @@ -40,7 +40,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ pkg-config \ && rm -rf /var/lib/apt/lists/* -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.88.0 +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.95.0 RUN rustup target add aarch64-apple-darwin @@ -57,35 +57,30 @@ RUN ln -sf /osxcross/bin/arm64-apple-darwin25.1-ld /usr/local/bin/arm64-apple-ma # --------------------------------------------------------------------------- # Stage 1: dependency caching — copy only manifests, create dummy sources, -# build dependencies. This layer is cached unless Cargo.toml/lock changes. +# build dependencies. This layer is cached unless Cargo.toml/lock changes. # --------------------------------------------------------------------------- COPY Cargo.toml Cargo.lock ./ -COPY crates/openshell-vm/Cargo.toml crates/openshell-vm/Cargo.toml -COPY crates/openshell-vm/build.rs crates/openshell-vm/build.rs -COPY crates/openshell-core/Cargo.toml crates/openshell-core/Cargo.toml -COPY crates/openshell-core/build.rs crates/openshell-core/build.rs -COPY crates/openshell-bootstrap/Cargo.toml crates/openshell-bootstrap/Cargo.toml -COPY crates/openshell-policy/Cargo.toml crates/openshell-policy/Cargo.toml +COPY crates/openshell-driver-vm/Cargo.toml crates/openshell-driver-vm/Cargo.toml +COPY crates/openshell-driver-vm/build.rs crates/openshell-driver-vm/build.rs +COPY crates/openshell-core/Cargo.toml crates/openshell-core/Cargo.toml +COPY crates/openshell-core/build.rs crates/openshell-core/build.rs COPY proto/ proto/ -# Scope workspace to VM crates only. -RUN sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-vm", "crates/openshell-core", "crates/openshell-bootstrap", "crates/openshell-policy"]|' Cargo.toml - -RUN mkdir -p crates/openshell-vm/src \ - crates/openshell-core/src \ - crates/openshell-bootstrap/src \ - crates/openshell-policy/src && \ - echo "fn main() {}" > crates/openshell-vm/src/main.rs && \ - touch crates/openshell-vm/src/lib.rs && \ - touch crates/openshell-core/src/lib.rs && \ - touch crates/openshell-bootstrap/src/lib.rs && \ - touch crates/openshell-policy/src/lib.rs - -# Build deps only (cached layer). -RUN --mount=type=cache,id=cargo-registry-vm-macos,sharing=locked,target=/root/.cargo/registry \ - --mount=type=cache,id=cargo-git-vm-macos,sharing=locked,target=/root/.cargo/git \ - --mount=type=cache,id=cargo-target-vm-macos-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ - cargo build --release --target aarch64-apple-darwin -p openshell-vm 2>/dev/null || true +# Scope workspace to the driver + its only internal dep. +RUN sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-driver-vm", "crates/openshell-core"]|' Cargo.toml + +RUN mkdir -p crates/openshell-driver-vm/src \ + crates/openshell-core/src && \ + echo "fn main() {}" > crates/openshell-driver-vm/src/main.rs && \ + touch crates/openshell-driver-vm/src/lib.rs && \ + touch crates/openshell-core/src/lib.rs + +# Build deps only (cached layer). The 2>/dev/null || true is a warm-cache +# technique; real source is copied in stage 2. +RUN --mount=type=cache,id=cargo-registry-driver-vm-macos,sharing=locked,target=/root/.cargo/registry \ + --mount=type=cache,id=cargo-git-driver-vm-macos,sharing=locked,target=/root/.cargo/git \ + --mount=type=cache,id=cargo-target-driver-vm-macos-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ + cargo build --release --target aarch64-apple-darwin -p openshell-driver-vm 2>/dev/null || true # --------------------------------------------------------------------------- # Stage 2: real build with compressed runtime artifacts @@ -97,29 +92,27 @@ COPY crates/ crates/ COPY --from=vm-runtime-compressed / /build/vm-runtime-compressed/ # Touch source files to ensure they're rebuilt (not the cached dummy). -RUN touch crates/openshell-vm/src/main.rs \ - crates/openshell-vm/src/lib.rs \ - crates/openshell-vm/build.rs \ - crates/openshell-bootstrap/src/lib.rs \ +RUN touch crates/openshell-driver-vm/src/main.rs \ + crates/openshell-driver-vm/src/lib.rs \ + crates/openshell-driver-vm/build.rs \ crates/openshell-core/src/lib.rs \ crates/openshell-core/build.rs \ - crates/openshell-policy/src/lib.rs \ proto/*.proto # Declare version ARGs here (not earlier) so the git-hash-bearing values do not # invalidate the expensive dependency-build layers above on every commit. ARG OPENSHELL_CARGO_VERSION ARG OPENSHELL_IMAGE_TAG -RUN --mount=type=cache,id=cargo-registry-vm-macos,sharing=locked,target=/root/.cargo/registry \ - --mount=type=cache,id=cargo-git-vm-macos,sharing=locked,target=/root/.cargo/git \ - --mount=type=cache,id=cargo-target-vm-macos-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ +RUN --mount=type=cache,id=cargo-registry-driver-vm-macos,sharing=locked,target=/root/.cargo/registry \ + --mount=type=cache,id=cargo-git-driver-vm-macos,sharing=locked,target=/root/.cargo/git \ + --mount=type=cache,id=cargo-target-driver-vm-macos-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ if [ -n "${OPENSHELL_CARGO_VERSION:-}" ]; then \ sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${OPENSHELL_CARGO_VERSION}"'"/}' Cargo.toml; \ fi && \ OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=/build/vm-runtime-compressed \ OPENSHELL_IMAGE_TAG="${OPENSHELL_IMAGE_TAG:-dev}" \ - cargo build --release --target aarch64-apple-darwin -p openshell-vm && \ - cp target/aarch64-apple-darwin/release/openshell-vm /openshell-vm + cargo build --release --target aarch64-apple-darwin -p openshell-driver-vm && \ + cp target/aarch64-apple-darwin/release/openshell-driver-vm /openshell-driver-vm FROM scratch AS binary -COPY --from=builder /openshell-vm /openshell-vm +COPY --from=builder /openshell-driver-vm /openshell-driver-vm diff --git a/deploy/docker/Dockerfile.gateway-macos b/deploy/docker/Dockerfile.gateway-macos new file mode 100644 index 0000000000..9d83c7990f --- /dev/null +++ b/deploy/docker/Dockerfile.gateway-macos @@ -0,0 +1,106 @@ +# syntax=docker/dockerfile:1.6 + +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Cross-compile the standalone openshell-gateway binary for macOS aarch64 +# (Apple Silicon) using the osxcross toolchain. + +ARG OSXCROSS_IMAGE=crazymax/osxcross:latest + +FROM ${OSXCROSS_IMAGE} AS osxcross + +FROM python:3.12-slim AS builder + +ARG CARGO_TARGET_CACHE_SCOPE=default + +ENV PATH="/root/.cargo/bin:/usr/local/bin:/osxcross/bin:${PATH}" +ENV LD_LIBRARY_PATH="/osxcross/lib" + +COPY --from=osxcross /osxcross /osxcross + +RUN SDKROOT="$(echo /osxcross/SDK/MacOSX*.sdk)" && ln -sfn "${SDKROOT}" /osxcross/SDK/MacOSX.sdk + +RUN apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + ca-certificates \ + clang \ + cmake \ + curl \ + libclang-dev \ + pkg-config \ + && rm -rf /var/lib/apt/lists/* + +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.95.0 + +RUN ln -sf /osxcross/bin/arm64-apple-darwin25.1-ld /usr/local/bin/arm64-apple-macosx-ld + +RUN rustup target add aarch64-apple-darwin + +WORKDIR /build + +ENV CC_aarch64_apple_darwin=oa64-clang +ENV CXX_aarch64_apple_darwin=oa64-clang++ +ENV AR_aarch64_apple_darwin=aarch64-apple-darwin25.1-ar +ENV CARGO_TARGET_AARCH64_APPLE_DARWIN_LINKER=oa64-clang +ENV CARGO_TARGET_AARCH64_APPLE_DARWIN_AR=aarch64-apple-darwin25.1-ar +ENV SDKROOT=/osxcross/SDK/MacOSX.sdk +ENV MACOSX_DEPLOYMENT_TARGET=13.3 +ENV CFLAGS_aarch64_apple_darwin=--target=arm64-apple-macosx\ -mmacosx-version-min=13.3 +ENV CXXFLAGS_aarch64_apple_darwin=--target=arm64-apple-macosx\ -mmacosx-version-min=13.3 +ENV BINDGEN_EXTRA_CLANG_ARGS_aarch64_apple_darwin=--target=arm64-apple-macosx\ -isysroot\ ${SDKROOT} + +COPY Cargo.toml Cargo.lock ./ +COPY crates/openshell-core/Cargo.toml crates/openshell-core/Cargo.toml +COPY crates/openshell-driver-kubernetes/Cargo.toml crates/openshell-driver-kubernetes/Cargo.toml +COPY crates/openshell-policy/Cargo.toml crates/openshell-policy/Cargo.toml +COPY crates/openshell-router/Cargo.toml crates/openshell-router/Cargo.toml +COPY crates/openshell-server/Cargo.toml crates/openshell-server/Cargo.toml +COPY crates/openshell-core/build.rs crates/openshell-core/build.rs +COPY proto/ proto/ + +RUN sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-server", "crates/openshell-core", "crates/openshell-driver-kubernetes", "crates/openshell-policy", "crates/openshell-router"]|' Cargo.toml + +RUN mkdir -p crates/openshell-core/src \ + crates/openshell-driver-kubernetes/src \ + crates/openshell-policy/src \ + crates/openshell-router/src \ + crates/openshell-server/src && \ + touch crates/openshell-core/src/lib.rs && \ + touch crates/openshell-driver-kubernetes/src/lib.rs && \ + printf 'fn main() {}\n' > crates/openshell-driver-kubernetes/src/main.rs && \ + touch crates/openshell-policy/src/lib.rs && \ + touch crates/openshell-router/src/lib.rs && \ + touch crates/openshell-server/src/lib.rs && \ + printf 'fn main() {}\n' > crates/openshell-server/src/main.rs + +RUN --mount=type=cache,id=cargo-registry-gateway-macos,sharing=locked,target=/root/.cargo/registry \ + --mount=type=cache,id=cargo-git-gateway-macos,sharing=locked,target=/root/.cargo/git \ + --mount=type=cache,id=cargo-target-gateway-macos-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ + cargo build --release --target aarch64-apple-darwin -p openshell-server 2>/dev/null || true + +COPY crates/ crates/ +COPY providers/ providers/ + +RUN touch crates/openshell-core/src/lib.rs \ + crates/openshell-driver-kubernetes/src/lib.rs \ + crates/openshell-driver-kubernetes/src/main.rs \ + crates/openshell-policy/src/lib.rs \ + crates/openshell-router/src/lib.rs \ + crates/openshell-server/src/lib.rs \ + crates/openshell-server/src/main.rs \ + crates/openshell-core/build.rs \ + proto/*.proto + +ARG OPENSHELL_CARGO_VERSION +RUN --mount=type=cache,id=cargo-registry-gateway-macos,sharing=locked,target=/root/.cargo/registry \ + --mount=type=cache,id=cargo-git-gateway-macos,sharing=locked,target=/root/.cargo/git \ + --mount=type=cache,id=cargo-target-gateway-macos-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ + if [ -n "${OPENSHELL_CARGO_VERSION:-}" ]; then \ + sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${OPENSHELL_CARGO_VERSION}"'"/}' Cargo.toml; \ + fi && \ + cargo build --release --target aarch64-apple-darwin -p openshell-server && \ + cp target/aarch64-apple-darwin/release/openshell-gateway /openshell-gateway + +FROM scratch AS binary +COPY --from=builder /openshell-gateway /openshell-gateway diff --git a/deploy/docker/Dockerfile.images b/deploy/docker/Dockerfile.images index b7e854677d..5172dc430d 100644 --- a/deploy/docker/Dockerfile.images +++ b/deploy/docker/Dockerfile.images @@ -7,10 +7,15 @@ # # Targets: # gateway Final gateway image +# supervisor Final supervisor image (FROM scratch, binary only) # cluster Final cluster image -# gateway-builder Release openshell-server binary -# supervisor-builder Release openshell-sandbox binary -# supervisor-output Minimal stage exporting only the supervisor binary +# +# Rust binaries are built natively before the image build and staged at: +# deploy/docker/.build/prebuilt-binaries//openshell-{gateway,sandbox} +# +# For local dev (Skaffold), pass --build-arg BUILD_FROM_SOURCE=1 to compile +# binaries inside Docker instead. BuildKit only executes the selected binary +# staging stage, so missing prebuilt files do not cause a build failure. # Pin by tag AND manifest-list digest to prevent silent upstream republishes # from breaking the build. Update both when bumping k3s versions. @@ -20,161 +25,67 @@ ARG K3S_DIGEST=sha256:4607083d3cac07e1ccde7317297271d13ed5f60f35a78f33fcef84858a ARG K9S_VERSION=v0.50.18 ARG HELM_VERSION=v3.17.3 ARG NVIDIA_CONTAINER_TOOLKIT_VERSION=1.18.2-1 +# Controls binary source: 0 = prebuilt (release), 1 = compile in Docker (local dev). +# Must be declared here (global scope) so it can be used in FROM instructions below. +ARG BUILD_FROM_SOURCE=0 # --------------------------------------------------------------------------- -# Shared Rust build stages +# Optional in-Docker Rust build (BUILD_FROM_SOURCE=1, local dev only) # --------------------------------------------------------------------------- -FROM --platform=$BUILDPLATFORM rust:1.88-slim AS rust-builder-base -ARG TARGETARCH -ARG BUILDARCH -ARG CARGO_TARGET_CACHE_SCOPE=default -ARG SCCACHE_MEMCACHED_ENDPOINT -# CI sets this to 1 for maximum optimization; local builds leave it unset -# so cargo uses the Cargo.toml default (parallel codegen for fast linking). -ARG CARGO_CODEGEN_UNITS +FROM rust:1.95.0-slim-bookworm AS rust-builder RUN apt-get update && apt-get install -y --no-install-recommends \ - cmake g++ make protobuf-compiler curl && rm -rf /var/lib/apt/lists/* - -COPY deploy/docker/cross-build.sh /usr/local/bin/ -RUN . cross-build.sh && install_cross_toolchain && install_sccache && add_rust_target + build-essential \ + cmake \ + pkg-config \ + libssl-dev \ + ca-certificates \ + && rm -rf /var/lib/apt/lists/* WORKDIR /build -FROM rust-builder-base AS rust-builder-skeleton - COPY Cargo.toml Cargo.lock ./ -COPY crates/openshell-bootstrap/Cargo.toml crates/openshell-bootstrap/Cargo.toml -COPY crates/openshell-cli/Cargo.toml crates/openshell-cli/Cargo.toml -COPY crates/openshell-core/Cargo.toml crates/openshell-core/Cargo.toml -COPY crates/openshell-driver-kubernetes/Cargo.toml crates/openshell-driver-kubernetes/Cargo.toml -COPY crates/openshell-ocsf/Cargo.toml crates/openshell-ocsf/Cargo.toml -COPY crates/openshell-policy/Cargo.toml crates/openshell-policy/Cargo.toml -COPY crates/openshell-providers/Cargo.toml crates/openshell-providers/Cargo.toml -COPY crates/openshell-prover/Cargo.toml crates/openshell-prover/Cargo.toml -COPY crates/openshell-router/Cargo.toml crates/openshell-router/Cargo.toml -COPY crates/openshell-sandbox/Cargo.toml crates/openshell-sandbox/Cargo.toml -COPY crates/openshell-server/Cargo.toml crates/openshell-server/Cargo.toml -COPY crates/openshell-tui/Cargo.toml crates/openshell-tui/Cargo.toml -COPY crates/openshell-vm/Cargo.toml crates/openshell-vm/Cargo.toml -COPY crates/openshell-core/build.rs crates/openshell-core/build.rs +COPY crates/ crates/ COPY proto/ proto/ - -RUN mkdir -p \ - crates/openshell-bootstrap/src \ - crates/openshell-cli/src \ - crates/openshell-core/src \ - crates/openshell-driver-kubernetes/src \ - crates/openshell-ocsf/src \ - crates/openshell-policy/src \ - crates/openshell-providers/src \ - crates/openshell-prover/src \ - crates/openshell-router/src \ - crates/openshell-sandbox/src \ - crates/openshell-server/src \ - crates/openshell-tui/src \ - crates/openshell-vm/src && \ - touch crates/openshell-bootstrap/src/lib.rs && \ - printf 'fn main() {}\n' > crates/openshell-cli/src/main.rs && \ - touch crates/openshell-core/src/lib.rs && \ - touch crates/openshell-driver-kubernetes/src/lib.rs && \ - printf 'fn main() {}\n' > crates/openshell-driver-kubernetes/src/main.rs && \ - touch crates/openshell-ocsf/src/lib.rs && \ - touch crates/openshell-policy/src/lib.rs && \ - touch crates/openshell-providers/src/lib.rs && \ - touch crates/openshell-prover/src/lib.rs && \ - touch crates/openshell-router/src/lib.rs && \ - touch crates/openshell-sandbox/src/lib.rs && \ - printf 'fn main() {}\n' > crates/openshell-sandbox/src/main.rs && \ - touch crates/openshell-server/src/lib.rs && \ - printf 'fn main() {}\n' > crates/openshell-server/src/main.rs && \ - touch crates/openshell-tui/src/lib.rs && \ - touch crates/openshell-vm/src/lib.rs && \ - printf 'fn main() {}\n' > crates/openshell-vm/src/main.rs - -FROM rust-builder-skeleton AS rust-deps - -RUN --mount=type=cache,id=cargo-registry-${TARGETARCH},sharing=locked,target=/usr/local/cargo/registry \ - --mount=type=cache,id=cargo-git-${TARGETARCH},sharing=locked,target=/usr/local/cargo/git \ - --mount=type=cache,id=cargo-target-${TARGETARCH}-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ - --mount=type=cache,id=sccache-${TARGETARCH},sharing=locked,target=/tmp/sccache \ - . cross-build.sh && cargo_cross_build --release -p openshell-server -p openshell-sandbox +COPY providers/ providers/ + +RUN --mount=type=cache,target=/usr/local/cargo/registry \ + --mount=type=cache,target=/build/target \ + cargo build --release \ + --features "openshell-core/dev-settings" \ + --bin openshell-gateway \ + --bin openshell-sandbox \ + && mkdir -p /build/out \ + && install -m 0755 target/release/openshell-gateway /build/out/openshell-gateway \ + && install -m 0755 target/release/openshell-sandbox /build/out/openshell-sandbox # --------------------------------------------------------------------------- -# Per-target workspace stages +# Per-arch binary stages # --------------------------------------------------------------------------- -# Copy only the crates needed for each target so that a change to -# openshell-sandbox does not invalidate the gateway build and vice versa. -# The skeleton stage already has stub Cargo.toml + src/ for every crate, -# so cargo workspace resolution continues to work — we just overwrite the -# crates whose real source is needed for compilation. - -FROM rust-deps AS gateway-workspace -ARG OPENSHELL_CARGO_VERSION - -COPY crates/openshell-core/ crates/openshell-core/ -COPY crates/openshell-driver-kubernetes/ crates/openshell-driver-kubernetes/ -COPY crates/openshell-policy/ crates/openshell-policy/ -COPY crates/openshell-providers/ crates/openshell-providers/ -COPY crates/openshell-router/ crates/openshell-router/ -COPY crates/openshell-server/ crates/openshell-server/ - -RUN touch \ - crates/openshell-core/build.rs \ - crates/openshell-server/src/main.rs \ - proto/*.proto && \ - if [ -n "${OPENSHELL_CARGO_VERSION:-}" ]; then \ - sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${OPENSHELL_CARGO_VERSION}"'"/}' Cargo.toml; \ - fi - -FROM gateway-workspace AS gateway-builder -ARG CARGO_CODEGEN_UNITS -ARG EXTRA_CARGO_FEATURES="" - -RUN --mount=type=cache,id=cargo-registry-${TARGETARCH},sharing=locked,target=/usr/local/cargo/registry \ - --mount=type=cache,id=cargo-git-${TARGETARCH},sharing=locked,target=/usr/local/cargo/git \ - --mount=type=cache,id=cargo-target-${TARGETARCH}-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ - --mount=type=cache,id=sccache-${TARGETARCH},sharing=locked,target=/tmp/sccache \ - . cross-build.sh && \ - cargo_cross_build --release -p openshell-server ${EXTRA_CARGO_FEATURES:+--features "$EXTRA_CARGO_FEATURES"} && \ - mkdir -p /build/out && \ - cp "$(cross_output_dir release)/openshell-server" /build/out/ - -FROM rust-deps AS supervisor-workspace -ARG OPENSHELL_CARGO_VERSION - -COPY crates/openshell-core/ crates/openshell-core/ -COPY crates/openshell-ocsf/ crates/openshell-ocsf/ -COPY crates/openshell-policy/ crates/openshell-policy/ -COPY crates/openshell-router/ crates/openshell-router/ -COPY crates/openshell-sandbox/ crates/openshell-sandbox/ - -RUN touch \ - crates/openshell-core/build.rs \ - crates/openshell-ocsf/src/lib.rs \ - crates/openshell-sandbox/src/main.rs \ - proto/*.proto && \ - if [ -n "${OPENSHELL_CARGO_VERSION:-}" ]; then \ - sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${OPENSHELL_CARGO_VERSION}"'"/}' Cargo.toml; \ - fi - -FROM supervisor-workspace AS supervisor-builder -ARG CARGO_CODEGEN_UNITS -ARG EXTRA_CARGO_FEATURES="" - -RUN --mount=type=cache,id=cargo-registry-${TARGETARCH},sharing=locked,target=/usr/local/cargo/registry \ - --mount=type=cache,id=cargo-git-${TARGETARCH},sharing=locked,target=/usr/local/cargo/git \ - --mount=type=cache,id=cargo-target-${TARGETARCH}-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ - --mount=type=cache,id=sccache-${TARGETARCH},sharing=locked,target=/tmp/sccache \ - . cross-build.sh && \ - cargo_cross_build --release -p openshell-sandbox ${EXTRA_CARGO_FEATURES:+--features "$EXTRA_CARGO_FEATURES"} && \ - mkdir -p /build/out && \ - cp "$(cross_output_dir release)/openshell-sandbox" /build/out/ - -# Minimal extraction stage for fast-deploy: exports only the supervisor -# binary (~20-40 MB) instead of the entire build environment (~968 MB). -FROM scratch AS supervisor-output -COPY --from=supervisor-builder /build/out/openshell-sandbox /openshell-sandbox + +# Prebuilt path (release default, BUILD_FROM_SOURCE=0) +FROM scratch AS gateway-binary-0 +ARG TARGETARCH +# --chmod=755 preserves the executable bit through actions/upload-artifact + +# download-artifact, which strip exec perms during the roundtrip. +COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-gateway /build/out/openshell-gateway + +# Source-built path (local dev, BUILD_FROM_SOURCE=1) +FROM rust-builder AS gateway-binary-1 + +FROM gateway-binary-${BUILD_FROM_SOURCE} AS gateway-binary + +# Prebuilt path (release default, BUILD_FROM_SOURCE=0) +FROM scratch AS supervisor-binary-0 +ARG TARGETARCH +# --chmod=755 preserves the executable bit through actions/upload-artifact + +# download-artifact, which strip exec perms during the roundtrip. +COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /build/out/openshell-sandbox + +# Source-built path (local dev, BUILD_FROM_SOURCE=1) +FROM rust-builder AS supervisor-binary-1 + +FROM supervisor-binary-${BUILD_FROM_SOURCE} AS supervisor-binary # --------------------------------------------------------------------------- # Final gateway image @@ -190,7 +101,7 @@ RUN useradd --create-home --user-group openshell WORKDIR /app -COPY --from=gateway-builder /build/out/openshell-server /usr/local/bin/ +COPY --from=gateway-binary /build/out/openshell-gateway /usr/local/bin/ RUN mkdir -p /build/crates/openshell-server COPY --chmod=755 crates/openshell-server/migrations /build/crates/openshell-server/migrations @@ -198,8 +109,17 @@ COPY --chmod=755 crates/openshell-server/migrations /build/crates/openshell-serv USER openshell EXPOSE 8080 -ENTRYPOINT ["openshell-server"] -CMD ["--port", "8080"] +ENTRYPOINT ["openshell-gateway"] +CMD ["--bind-address", "0.0.0.0", "--port", "8080"] + +# --------------------------------------------------------------------------- +# Final supervisor image +# --------------------------------------------------------------------------- +# Minimal FROM scratch image containing only the supervisor binary. +# Used by both the Docker driver (binary extraction) and the Podman driver +# (OCI image volume mount at /opt/openshell/bin). +FROM scratch AS supervisor +COPY --from=supervisor-binary /build/out/openshell-sandbox /openshell-sandbox # --------------------------------------------------------------------------- # Cluster asset stages @@ -265,7 +185,7 @@ COPY --from=nvidia-container-toolkit /usr/bin/nvidia-cdi-hook /usr/bin/ COPY --from=nvidia-container-toolkit /usr/bin/nvidia-container-runtime /usr/bin/ COPY --from=nvidia-container-toolkit /usr/bin/nvidia-ctk /usr/bin/ COPY --from=nvidia-container-toolkit /etc/nvidia-container-runtime /etc/nvidia-container-runtime -COPY --from=supervisor-builder /build/out/openshell-sandbox /opt/openshell/bin/openshell-sandbox +COPY --from=supervisor-binary /build/out/openshell-sandbox /opt/openshell/bin/openshell-sandbox RUN mkdir -p /var/lib/rancher/k3s/server/manifests \ /var/lib/rancher/k3s/server/static/charts \ @@ -282,7 +202,13 @@ COPY deploy/docker/cluster-healthcheck.sh /usr/local/bin/cluster-healthcheck.sh RUN chmod +x /usr/local/bin/cluster-healthcheck.sh COPY deploy/docker/.build/charts/*.tgz /opt/openshell/charts/ -COPY deploy/kube/manifests/*.yaml /opt/openshell/manifests/ +# Only the core k3s auto-deploy manifests belong in the cluster image. +# Gateway API routing is optional and requires Envoy Gateway CRDs, so +# deploy/kube/manifests/envoy-gateway-openshell.yaml stays repo-local and is +# applied manually by `mise run helm:gateway:apply` when grpcRoute is enabled. +COPY deploy/kube/manifests/openshell-helmchart.yaml \ + deploy/kube/manifests/agent-sandbox.yaml \ + /opt/openshell/manifests/ COPY deploy/kube/gpu-manifests/*.yaml /opt/openshell/gpu-manifests/ ENTRYPOINT ["/usr/local/bin/cluster-entrypoint.sh"] diff --git a/deploy/docker/Dockerfile.python-wheels b/deploy/docker/Dockerfile.python-wheels index 91e2223c47..e93bf8f22e 100644 --- a/deploy/docker/Dockerfile.python-wheels +++ b/deploy/docker/Dockerfile.python-wheels @@ -21,7 +21,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ libssl-dev \ && rm -rf /var/lib/apt/lists/* -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.95.0 RUN pip install --no-cache-dir maturin COPY deploy/docker/cross-build.sh /usr/local/bin/ diff --git a/deploy/docker/Dockerfile.python-wheels-macos b/deploy/docker/Dockerfile.python-wheels-macos index 79cc6d9b4d..8c22537e95 100644 --- a/deploy/docker/Dockerfile.python-wheels-macos +++ b/deploy/docker/Dockerfile.python-wheels-macos @@ -36,7 +36,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ # arm64-apple-macosx-ld. Provide a linker alias to osxcross ld64. RUN ln -sf /osxcross/bin/arm64-apple-darwin25.1-ld /usr/local/bin/arm64-apple-macosx-ld -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.95.0 RUN rustup target add aarch64-apple-darwin RUN pip install --no-cache-dir maturin @@ -91,6 +91,7 @@ RUN --mount=type=cache,id=cargo-registry-python-wheels-macos-${TARGETARCH},shari # Copy actual source code and Python packaging files. COPY crates/ crates/ +COPY providers/ providers/ COPY pyproject.toml README.md ./ COPY python/ python/ diff --git a/deploy/docker/cluster-entrypoint.sh b/deploy/docker/cluster-entrypoint.sh index b045bf2224..56e577e7f7 100644 --- a/deploy/docker/cluster-entrypoint.sh +++ b/deploy/docker/cluster-entrypoint.sh @@ -431,15 +431,17 @@ fi # In push mode, use the exact image references that were imported into cluster # containerd so the Helm release cannot drift back to remote ":latest" tags. -# Only the gateway image is pushed; sandbox images are pulled from the -# community registry at runtime. +# Gateway and supervisor images may be pushed; sandbox base images are pulled +# from the community registry at runtime. if [ -n "${PUSH_IMAGE_REFS:-}" ] && [ -f "$HELMCHART" ]; then server_image="" + supervisor_image="" old_ifs="$IFS" IFS=',' for ref in $PUSH_IMAGE_REFS; do case "$ref" in */gateway:*) server_image="$ref" ;; + */supervisor:*) supervisor_image="$ref" ;; esac done IFS="$old_ifs" @@ -452,13 +454,20 @@ if [ -n "${PUSH_IMAGE_REFS:-}" ] && [ -f "$HELMCHART" ]; then sed -i -E "s|repository:[[:space:]]*[^[:space:]]+|repository: ${server_repo}|" "$HELMCHART" sed -i -E "s|tag:[[:space:]]*\"?[^\"[:space:]]+\"?|tag: \"${server_tag}\"|" "$HELMCHART" fi + + if [ -n "$supervisor_image" ]; then + echo "Setting supervisor image: ${supervisor_image}" + sed -i -E "s|supervisorImage:[[:space:]]*\"?[^\"]+\"?|supervisorImage: ${supervisor_image}|" "$HELMCHART" + fi fi if [ -n "${IMAGE_TAG:-}" ] && [ -f "$HELMCHART" ]; then - echo "Overriding gateway image tag to: ${IMAGE_TAG}" + echo "Overriding gateway and supervisor image tags to: ${IMAGE_TAG}" # server image tag (standalone value field) # Handle both quoted and unquoted defaults: tag: "latest" / tag: latest sed -i -E "s|tag:[[:space:]]*\"?latest\"?|tag: \"${IMAGE_TAG}\"|" "$HELMCHART" + # supervisor image is a full image ref under server.supervisorImage + sed -i -E "s|(supervisorImage:[[:space:]]*\"?[^\"]*:)[^\"[:space:]]+(\"?)|\\1${IMAGE_TAG}\\2|" "$HELMCHART" fi if [ -f "$HELMCHART" ]; then @@ -506,6 +515,25 @@ if [ -f "$HELMCHART" ]; then sed -i "s|__DISABLE_GATEWAY_AUTH__|false|g" "$HELMCHART" fi + # OIDC JWT authentication: when OIDC_ISSUER is set, the server validates + # Bearer tokens on gRPC requests against the issuer's JWKS endpoint. + if [ -n "${OIDC_ISSUER:-}" ]; then + echo "Enabling OIDC authentication (issuer: ${OIDC_ISSUER})" + sed -i "s|__OIDC_ISSUER__|${OIDC_ISSUER}|g" "$HELMCHART" + sed -i "s|__OIDC_AUDIENCE__|${OIDC_AUDIENCE:-openshell-cli}|g" "$HELMCHART" + sed -i "s|__OIDC_ROLES_CLAIM__|${OIDC_ROLES_CLAIM:-realm_access.roles}|g" "$HELMCHART" + sed -i "s|__OIDC_ADMIN_ROLE__|${OIDC_ADMIN_ROLE:-openshell-admin}|g" "$HELMCHART" + sed -i "s|__OIDC_USER_ROLE__|${OIDC_USER_ROLE:-openshell-user}|g" "$HELMCHART" + sed -i "s|__OIDC_SCOPES_CLAIM__|${OIDC_SCOPES_CLAIM:-}|g" "$HELMCHART" + else + sed -i "s|__OIDC_ISSUER__||g" "$HELMCHART" + sed -i "s|__OIDC_AUDIENCE__|openshell-cli|g" "$HELMCHART" + sed -i "s|__OIDC_ROLES_CLAIM__||g" "$HELMCHART" + sed -i "s|__OIDC_ADMIN_ROLE__||g" "$HELMCHART" + sed -i "s|__OIDC_USER_ROLE__||g" "$HELMCHART" + sed -i "s|__OIDC_SCOPES_CLAIM__||g" "$HELMCHART" + fi + # Disable TLS entirely: the server listens on plaintext HTTP. # Used when a reverse proxy / tunnel terminates TLS at the edge. if [ "${DISABLE_TLS:-}" = "true" ]; then diff --git a/deploy/docker/cluster-healthcheck.sh b/deploy/docker/cluster-healthcheck.sh index e2828c6e54..96c3264464 100644 --- a/deploy/docker/cluster-healthcheck.sh +++ b/deploy/docker/cluster-healthcheck.sh @@ -54,8 +54,8 @@ kubectl -n openshell wait --for=jsonpath='{.status.readyReplicas}'=1 statefulset # Verify the sandbox supervisor binary exists on the node filesystem. # Sandbox pods mount /opt/openshell/bin as a read-only hostPath volume and # exec /opt/openshell/bin/openshell-sandbox as their entrypoint. If the binary -# is missing (e.g. cluster image was built without the supervisor-builder -# stage), every sandbox pod will crash with "no such file or directory". +# is missing (e.g. cluster image was built without the staged prebuilt +# binary), every sandbox pod will crash with "no such file or directory". # --------------------------------------------------------------------------- if [ ! -x /opt/openshell/bin/openshell-sandbox ]; then echo "HEALTHCHECK_MISSING_SUPERVISOR: /opt/openshell/bin/openshell-sandbox not found" >&2 @@ -78,5 +78,7 @@ kubectl -n openshell get secret openshell-ssh-handshake >/dev/null 2>&1 || exit # iptables rules for NodePort routing. Without this check the health check # can pass before the port is routable, causing "Connection refused" on the # host-mapped port. +# Use /dev/tcp/127.0.0.1/30051' 2>/dev/null || exit 1 +timeout 2 bash -c 'exec 3<>/dev/tcp/127.0.0.1/30051 && exec 3>&-' 2>/dev/null || exit 1 diff --git a/deploy/helm/openshell/.helmignore b/deploy/helm/openshell/.helmignore index 414bb6e8a6..798d0e7c84 100644 --- a/deploy/helm/openshell/.helmignore +++ b/deploy/helm/openshell/.helmignore @@ -16,3 +16,11 @@ .idea/ *.tmproj .vscode/ + +# Ignore development files +skaffold.yaml +values-keycloak.yaml +values-ingress.yaml +values-gateway.yaml +values-cert-manager.yaml +values-skaffold.yaml diff --git a/deploy/helm/openshell/Chart.yaml b/deploy/helm/openshell/Chart.yaml index f97e316cfd..b66bb6d015 100644 --- a/deploy/helm/openshell/Chart.yaml +++ b/deploy/helm/openshell/Chart.yaml @@ -5,5 +5,8 @@ apiVersion: v2 name: openshell description: runtime environment for autonomous agents type: application -version: 0.1.0 -appVersion: "0.1.0" +# Updated to the release version by CI. The appVersion doubles as the default +# image tag (image.tag defaults to appVersion when empty), so a released chart +# automatically pulls the matching gateway and supervisor images. +version: 0.0.0 +appVersion: "0.0.0" diff --git a/deploy/helm/openshell/skaffold.yaml b/deploy/helm/openshell/skaffold.yaml new file mode 100644 index 0000000000..fe7b96cf22 --- /dev/null +++ b/deploy/helm/openshell/skaffold.yaml @@ -0,0 +1,104 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Local dev: builds gateway + supervisor images using Dockerfile.images with +# BUILD_FROM_SOURCE=1, which compiles Rust binaries inside Docker without +# requiring pre-staged artifacts. +# +# Run from repo root: +# skaffold dev -f deploy/helm/openshell/skaffold.yaml +# +# See https://skaffold.dev/docs/deployers/helm/ (setValueTemplates, IMAGE_* fields). +apiVersion: skaffold/v4beta14 +kind: Config +metadata: + name: openshell +build: + local: + push: false + tagPolicy: + gitCommit: {} + artifacts: + - image: openshell/gateway + context: ../../.. + custom: + buildCommand: | + docker buildx build \ + --build-arg BUILD_FROM_SOURCE=1 \ + --target gateway \ + --tag "$IMAGE" \ + --load \ + --file deploy/docker/Dockerfile.images \ + . + dependencies: + paths: + - Cargo.toml + - Cargo.lock + - crates/** + - proto/** + - deploy/docker/Dockerfile.images + - crates/openshell-server/migrations/** + - image: openshell/supervisor + context: ../../.. + custom: + buildCommand: | + docker buildx build \ + --build-arg BUILD_FROM_SOURCE=1 \ + --target supervisor \ + --tag "$IMAGE" \ + --load \ + --file deploy/docker/Dockerfile.images \ + . + dependencies: + paths: + - Cargo.toml + - Cargo.lock + - crates/** + - proto/** + - deploy/docker/Dockerfile.images +deploy: + helm: + releases: + # cert-manager — comment this in and add values-cert-manager.yaml below + # when you want cert-manager to manage the PKI instead of pkiInitJob. + # Requires cert-manager CRDs to be installed in the cluster first. + #- name: cert-manager + # repo: https://charts.jetstack.io + # remoteChart: cert-manager + # version: v1.20.2 + # namespace: cert-manager + # createNamespace: true + # setValues: + # crds.enabled: true + # Envoy Gateway — Kubernetes Gateway API implementation. + # Installs the Gateway API CRDs and the "eg" GatewayClass. + # Required when grpcRoute.enabled is true in the openshell release. + #- name: envoy-gateway + # remoteChart: oci://docker.io/envoyproxy/gateway-helm + # version: v1.7.2 + # namespace: envoy-gateway-system + # createNamespace: true + # # wait ensures Gateway API CRDs are registered before the openshell + # # release attempts to create Gateway and HTTPRoute resources. + # wait: true + - name: openshell + chartPath: . + namespace: openshell + createNamespace: true + valuesFiles: + - values.yaml + - values-skaffold.yaml + # Add values-cert-manager.yaml here (and uncomment the cert-manager + # release above) to switch from pkiInitJob to cert-manager for PKI. + #- values-cert-manager.yaml + # To enable OIDC with a local Keycloak instance, run the one-time + # setup task first, then uncomment the line below: + # mise run keycloak:k8s:setup + #- values-keycloak.yaml + # To enable the Gateway API HTTPRoute (requires Envoy Gateway above): + #- values-gateway.yaml + setValueTemplates: + image.repository: '{{.IMAGE_REPO_openshell_gateway}}' + image.tag: '{{.IMAGE_TAG_openshell_gateway}}' + supervisor.image.repository: '{{.IMAGE_REPO_openshell_supervisor}}' + supervisor.image.tag: '{{.IMAGE_TAG_openshell_supervisor}}' diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 10fd3de0d2..09159340d4 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -58,3 +58,26 @@ Create the name of the service account to use {{- default "default" .Values.serviceAccount.name }} {{- end }} {{- end }} + +{{/* +Gateway image reference. Uses image.tag when set; falls back to .Chart.AppVersion +so a released chart automatically pulls the matching image without extra overrides. +*/}} +{{- define "openshell.image" -}} +{{- printf "%s:%s" .Values.image.repository (.Values.image.tag | default .Chart.AppVersion) }} +{{- end }} + +{{/* +Supervisor image reference. Same appVersion fallback as openshell.image so +the supervisor and gateway images stay in sync across releases. +*/}} +{{- define "openshell.supervisorImage" -}} +{{- printf "%s:%s" .Values.supervisor.image.repository (.Values.supervisor.image.tag | default .Chart.AppVersion) }} +{{- end }} + +{{/* +Namespaced Issuer (selfSigned) for cert-manager CA bootstrap. +*/}} +{{- define "openshell.issuerSelfSigned" -}} +{{- printf "%s-selfsigned" (include "openshell.fullname" .) | trunc 63 | trimSuffix "-" }} +{{- end }} diff --git a/deploy/helm/openshell/templates/cert-manager-pki.yaml b/deploy/helm/openshell/templates/cert-manager-pki.yaml new file mode 100644 index 0000000000..43a19d5f55 --- /dev/null +++ b/deploy/helm/openshell/templates/cert-manager-pki.yaml @@ -0,0 +1,98 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.certManager.enabled }} +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: {{ include "openshell.issuerSelfSigned" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + selfSigned: {} +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: {{ include "openshell.fullname" . }}-ca + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + isCA: true + commonName: openshell-ca + secretName: {{ .Values.certManager.caSecretName | quote }} + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: {{ include "openshell.issuerSelfSigned" . }} + kind: Issuer + group: cert-manager.io +--- +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: {{ include "openshell.fullname" . }}-ca-issuer + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + ca: + secretName: {{ .Values.certManager.caSecretName | quote }} +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: {{ include "openshell.fullname" . }}-server + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + secretName: {{ .Values.server.tls.certSecretName | quote }} + duration: {{ .Values.certManager.certificateDuration | quote }} + renewBefore: {{ .Values.certManager.certificateRenewBefore | quote }} + commonName: openshell-server + dnsNames: + {{- toYaml .Values.certManager.serverDnsNames | nindent 4 }} + {{- if .Values.certManager.serverIpAddresses }} + ipAddresses: + {{- toYaml .Values.certManager.serverIpAddresses | nindent 4 }} + {{- end }} + privateKey: + algorithm: ECDSA + size: 256 + usages: + - server auth + - digital signature + - key encipherment + issuerRef: + name: {{ include "openshell.fullname" . }}-ca-issuer + kind: Issuer + group: cert-manager.io +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: {{ include "openshell.fullname" . }}-client + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + secretName: {{ .Values.server.tls.clientTlsSecretName | quote }} + duration: {{ .Values.certManager.certificateDuration | quote }} + renewBefore: {{ .Values.certManager.certificateRenewBefore | quote }} + commonName: openshell-client + privateKey: + algorithm: ECDSA + size: 256 + usages: + - client auth + - digital signature + - key encipherment + issuerRef: + name: {{ include "openshell.fullname" . }}-ca-issuer + kind: Issuer + group: cert-manager.io +{{- end }} diff --git a/deploy/helm/openshell/templates/clusterrole.yaml b/deploy/helm/openshell/templates/clusterrole.yaml index 2f98784583..a660aee753 100644 --- a/deploy/helm/openshell/templates/clusterrole.yaml +++ b/deploy/helm/openshell/templates/clusterrole.yaml @@ -4,17 +4,10 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: - name: {{ include "openshell.fullname" . }}-sandbox-runtimeclass + name: {{ include "openshell.fullname" . }}-node-reader labels: {{- include "openshell.labels" . | nindent 4 }} rules: - - apiGroups: - - node.k8s.io - resources: - - runtimeclasses - verbs: - - get - - list - apiGroups: - "" resources: @@ -22,3 +15,4 @@ rules: verbs: - get - list + - watch diff --git a/deploy/helm/openshell/templates/clusterrolebinding.yaml b/deploy/helm/openshell/templates/clusterrolebinding.yaml index 9dda1d7da4..685a73bf91 100644 --- a/deploy/helm/openshell/templates/clusterrolebinding.yaml +++ b/deploy/helm/openshell/templates/clusterrolebinding.yaml @@ -4,13 +4,13 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: {{ include "openshell.fullname" . }}-sandbox-runtimeclass + name: {{ include "openshell.fullname" . }}-node-reader labels: {{- include "openshell.labels" . | nindent 4 }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole - name: {{ include "openshell.fullname" . }}-sandbox-runtimeclass + name: {{ include "openshell.fullname" . }}-node-reader subjects: - kind: ServiceAccount name: {{ include "openshell.serviceAccountName" . }} diff --git a/deploy/helm/openshell/templates/gateway.yaml b/deploy/helm/openshell/templates/gateway.yaml new file mode 100644 index 0000000000..3fd81345ad --- /dev/null +++ b/deploy/helm/openshell/templates/gateway.yaml @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if and .Values.grpcRoute.enabled .Values.grpcRoute.gateway.create }} +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: {{ default (include "openshell.fullname" .) .Values.grpcRoute.gateway.name }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + gatewayClassName: {{ .Values.grpcRoute.gateway.className }} + listeners: + - name: http + port: {{ .Values.grpcRoute.gateway.listener.port }} + protocol: {{ .Values.grpcRoute.gateway.listener.protocol }} + allowedRoutes: + namespaces: + from: {{ .Values.grpcRoute.gateway.listener.allowedRoutes }} +{{- end }} diff --git a/deploy/helm/openshell/templates/grpcroute.yaml b/deploy/helm/openshell/templates/grpcroute.yaml new file mode 100644 index 0000000000..8fde5458cd --- /dev/null +++ b/deploy/helm/openshell/templates/grpcroute.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.grpcRoute.enabled }} +apiVersion: gateway.networking.k8s.io/v1 +kind: GRPCRoute +metadata: + name: {{ include "openshell.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + parentRefs: + - name: {{ default (include "openshell.fullname" .) .Values.grpcRoute.gateway.name }} + namespace: {{ default .Release.Namespace .Values.grpcRoute.gateway.namespace }} + {{- if .Values.grpcRoute.hostnames }} + hostnames: + {{- toYaml .Values.grpcRoute.hostnames | nindent 4 }} + {{- end }} + rules: + - backendRefs: + - name: {{ include "openshell.fullname" . }} + port: {{ .Values.service.port }} +{{- end }} diff --git a/deploy/helm/openshell/templates/pki-hook.yaml b/deploy/helm/openshell/templates/pki-hook.yaml new file mode 100644 index 0000000000..c5e83c734a --- /dev/null +++ b/deploy/helm/openshell/templates/pki-hook.yaml @@ -0,0 +1,191 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if and .Values.pkiInitJob.enabled .Values.certManager.enabled }} +{{- fail "pkiInitJob.enabled and certManager.enabled cannot both be true; disable one to avoid conflicting PKI sources." }} +{{- end }} +{{- if .Values.pkiInitJob.enabled }} +{{- $hookName := printf "%s-pki-hook" (include "openshell.fullname" .) }} +{{- $ns := .Release.Namespace }} +{{- $serverSecret := .Values.server.tls.certSecretName }} +{{- $clientSecret := .Values.server.tls.clientTlsSecretName }} +{{- $sanParts := list }} +{{- range .Values.pkiInitJob.serverDnsNames }}{{- $sanParts = append $sanParts (printf "DNS:%s" .) }}{{- end }} +{{- range .Values.pkiInitJob.serverIpAddresses }}{{- $sanParts = append $sanParts (printf "IP:%s" .) }}{{- end }} +{{- $serverSans := join "," $sanParts }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ $hookName }} + namespace: {{ $ns }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" + helm.sh/hook-delete-policy: before-hook-creation +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ $hookName }} + namespace: {{ $ns }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" + helm.sh/hook-delete-policy: before-hook-creation +rules: + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ $hookName }} + namespace: {{ $ns }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" + helm.sh/hook-delete-policy: before-hook-creation +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ $hookName }} +subjects: + - kind: ServiceAccount + name: {{ $hookName }} + namespace: {{ $ns }} +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ $hookName }} + namespace: {{ $ns }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-20" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + backoffLimit: 3 + activeDeadlineSeconds: 120 + ttlSecondsAfterFinished: 300 + template: + metadata: + labels: + {{- include "openshell.selectorLabels" . | nindent 8 }} + spec: + restartPolicy: OnFailure + serviceAccountName: {{ $hookName }} + containers: + - name: pki-gen + image: {{ .Values.pkiInitJob.image.repository }}:{{ .Values.pkiInitJob.image.tag }} + imagePullPolicy: {{ .Values.pkiInitJob.image.pullPolicy }} + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + env: + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: SERVER_SECRET + value: {{ $serverSecret | quote }} + - name: CLIENT_SECRET + value: {{ $clientSecret | quote }} + - name: CA_DAYS + value: {{ .Values.pkiInitJob.caValidityDays | quote }} + - name: CERT_DAYS + value: {{ .Values.pkiInitJob.certValidityDays | quote }} + - name: SERVER_SANS + value: {{ $serverSans | quote }} + command: + - /bin/sh + - -c + - | + set -eu + apk add --no-cache openssl curl >/dev/null 2>&1 + + TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) + K8S_CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt + API=https://kubernetes.default.svc + + # Idempotency: skip only when both TLS secrets already exist. + # Checking one is insufficient — a partial cleanup can leave one half + # of the pair behind, which would cause mTLS to fail at runtime. + HTTP_SERVER=$(curl -s -o /dev/null -w "%{http_code}" \ + -H "Authorization: Bearer $TOKEN" --cacert "$K8S_CA" \ + "$API/api/v1/namespaces/$NAMESPACE/secrets/$SERVER_SECRET") + HTTP_CLIENT=$(curl -s -o /dev/null -w "%{http_code}" \ + -H "Authorization: Bearer $TOKEN" --cacert "$K8S_CA" \ + "$API/api/v1/namespaces/$NAMESPACE/secrets/$CLIENT_SECRET") + if [ "$HTTP_SERVER" = "200" ] && [ "$HTTP_CLIENT" = "200" ]; then + echo "PKI secrets already exist, skipping." + exit 0 + fi + if [ "$HTTP_SERVER" = "200" ] || [ "$HTTP_CLIENT" = "200" ]; then + echo "ERROR: partial PKI state — one secret exists but not both." >&2 + echo "To recover: kubectl delete secret -n $NAMESPACE $SERVER_SECRET $CLIENT_SECRET" >&2 + exit 1 + fi + + cd /tmp + + # CA (ECDSA P-256) + openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out ca.key 2>/dev/null + openssl req -new -x509 -sha256 -key ca.key -out ca.crt \ + -days "$CA_DAYS" -subj "/O=openshell/CN=openshell-ca" \ + -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \ + -addext "keyUsage=critical,keyCertSign,cRLSign" + + # Server cert (ECDSA P-256) + printf "[ext]\nsubjectAltName=%s\nextendedKeyUsage=serverAuth\nkeyUsage=digitalSignature,keyEncipherment\n" \ + "$SERVER_SANS" > server.ext + openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out server.key 2>/dev/null + openssl req -new -sha256 -key server.key -out server.csr -subj "/CN=openshell-server" + openssl x509 -req -sha256 -in server.csr -CA ca.crt -CAkey ca.key \ + -CAcreateserial -days "$CERT_DAYS" -extensions ext -extfile server.ext -out server.crt + + # Client cert (ECDSA P-256) + printf "[ext]\nextendedKeyUsage=clientAuth\nkeyUsage=digitalSignature,keyEncipherment\n" \ + > client.ext + openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out client.key 2>/dev/null + openssl req -new -sha256 -key client.key -out client.csr -subj "/CN=openshell-client" + openssl x509 -req -sha256 -in client.csr -CA ca.crt -CAkey ca.key \ + -CAcreateserial -days "$CERT_DAYS" -extensions ext -extfile client.ext -out client.crt + + CA_B64=$(base64 -w0 ca.crt) + SERVER_CRT_B64=$(base64 -w0 server.crt) + SERVER_KEY_B64=$(base64 -w0 server.key) + CLIENT_CRT_B64=$(base64 -w0 client.crt) + CLIENT_KEY_B64=$(base64 -w0 client.key) + + # Create server TLS secret + printf '{"apiVersion":"v1","kind":"Secret","metadata":{"name":"%s","namespace":"%s"},"type":"kubernetes.io/tls","data":{"tls.crt":"%s","tls.key":"%s","ca.crt":"%s"}}\n' \ + "$SERVER_SECRET" "$NAMESPACE" \ + "$SERVER_CRT_B64" "$SERVER_KEY_B64" "$CA_B64" > server-secret.json + curl -sf -X POST \ + -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ + --cacert "$K8S_CA" "$API/api/v1/namespaces/$NAMESPACE/secrets" \ + -d @server-secret.json + + # Create client TLS secret + printf '{"apiVersion":"v1","kind":"Secret","metadata":{"name":"%s","namespace":"%s"},"type":"kubernetes.io/tls","data":{"tls.crt":"%s","tls.key":"%s","ca.crt":"%s"}}\n' \ + "$CLIENT_SECRET" "$NAMESPACE" \ + "$CLIENT_CRT_B64" "$CLIENT_KEY_B64" "$CA_B64" > client-secret.json + curl -sf -X POST \ + -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ + --cacert "$K8S_CA" "$API/api/v1/namespaces/$NAMESPACE/secrets" \ + -d @client-secret.json + + rm -f *.key *.csr *.crt *.ext *.srl *.json + echo "PKI secrets created." +{{- end }} diff --git a/deploy/helm/openshell/templates/service.yaml b/deploy/helm/openshell/templates/service.yaml index 0e4aa4b763..ebad42eab1 100644 --- a/deploy/helm/openshell/templates/service.yaml +++ b/deploy/helm/openshell/templates/service.yaml @@ -18,5 +18,11 @@ spec: {{- if and (eq .Values.service.type "NodePort") .Values.service.nodePort }} nodePort: {{ .Values.service.nodePort }} {{- end }} + {{- if .Values.service.metricsPort }} + - port: {{ .Values.service.metricsPort }} + targetPort: metrics + protocol: TCP + name: metrics + {{- end }} selector: {{- include "openshell.selectorLabels" . | nindent 4 }} diff --git a/deploy/helm/openshell/templates/ssh-handshake-secret-hook.yaml b/deploy/helm/openshell/templates/ssh-handshake-secret-hook.yaml new file mode 100644 index 0000000000..ad444847b1 --- /dev/null +++ b/deploy/helm/openshell/templates/ssh-handshake-secret-hook.yaml @@ -0,0 +1,27 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if .Values.sshHandshake.hook.enabled }} +{{- $name := .Values.server.sshHandshakeSecretName }} +{{- $ns := .Release.Namespace }} +{{- $existing := lookup "v1" "Secret" $ns $name }} +{{- if not $existing }} +{{- $hex := .Values.sshHandshake.value }} +{{- if not $hex }} +{{- $hex = printf "%s%s" (uuidv4 | replace "-" "") (uuidv4 | replace "-" "") }} +{{- end }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ $name }} + namespace: {{ $ns }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-20" +type: Opaque +stringData: + secret: {{ $hex | quote }} +{{- end }} +{{- end }} diff --git a/deploy/helm/openshell/templates/statefulset.yaml b/deploy/helm/openshell/templates/statefulset.yaml index ed503a78e7..2db3a0c5f5 100644 --- a/deploy/helm/openshell/templates/statefulset.yaml +++ b/deploy/helm/openshell/templates/statefulset.yaml @@ -44,11 +44,19 @@ spec: - name: {{ .Chart.Name }} securityContext: {{- toYaml .Values.securityContext | nindent 12 }} - image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" + image: {{ include "openshell.image" . | quote }} imagePullPolicy: {{ .Values.image.pullPolicy }} args: + - --bind-address + - "0.0.0.0" - --port - {{ .Values.service.port | quote }} + - --health-port + - {{ .Values.service.healthPort | quote }} + {{- if .Values.service.metricsPort }} + - --metrics-port + - {{ .Values.service.metricsPort | quote }} + {{- end }} - --log-level - {{ .Values.server.logLevel }} - --db-url @@ -62,6 +70,12 @@ spec: - name: OPENSHELL_SANDBOX_IMAGE_PULL_POLICY value: {{ .Values.server.sandboxImagePullPolicy | quote }} {{- end }} + - name: OPENSHELL_SUPERVISOR_IMAGE + value: {{ include "openshell.supervisorImage" . | quote }} + {{- if .Values.supervisor.image.pullPolicy }} + - name: OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY + value: {{ .Values.supervisor.image.pullPolicy | quote }} + {{- end }} - name: OPENSHELL_GRPC_ENDPOINT value: {{ if .Values.server.disableTls }}{{ .Values.server.grpcEndpoint | replace "https://" "http://" | quote }}{{ else }}{{ .Values.server.grpcEndpoint | quote }}{{ end }} {{- if .Values.server.sshGatewayHost }} @@ -98,6 +112,30 @@ spec: value: "true" {{- end }} {{- end }} + {{- if .Values.server.oidc.issuer }} + - name: OPENSHELL_OIDC_ISSUER + value: {{ .Values.server.oidc.issuer | quote }} + - name: OPENSHELL_OIDC_AUDIENCE + value: {{ .Values.server.oidc.audience | quote }} + - name: OPENSHELL_OIDC_JWKS_TTL + value: {{ .Values.server.oidc.jwksTtl | quote }} + {{- if .Values.server.oidc.rolesClaim }} + - name: OPENSHELL_OIDC_ROLES_CLAIM + value: {{ .Values.server.oidc.rolesClaim | quote }} + {{- end }} + {{- if .Values.server.oidc.adminRole }} + - name: OPENSHELL_OIDC_ADMIN_ROLE + value: {{ .Values.server.oidc.adminRole | quote }} + {{- end }} + {{- if .Values.server.oidc.userRole }} + - name: OPENSHELL_OIDC_USER_ROLE + value: {{ .Values.server.oidc.userRole | quote }} + {{- end }} + {{- if .Values.server.oidc.scopesClaim }} + - name: OPENSHELL_OIDC_SCOPES_CLAIM + value: {{ .Values.server.oidc.scopesClaim | quote }} + {{- end }} + {{- end }} volumeMounts: - name: openshell-data mountPath: /var/openshell @@ -113,22 +151,33 @@ spec: - name: grpc containerPort: {{ .Values.service.port }} protocol: TCP + - name: health + containerPort: {{ .Values.service.healthPort }} + protocol: TCP + {{- if .Values.service.metricsPort }} + - name: metrics + containerPort: {{ .Values.service.metricsPort }} + protocol: TCP + {{- end }} startupProbe: - tcpSocket: - port: grpc + httpGet: + path: /healthz + port: health periodSeconds: {{ .Values.probes.startup.periodSeconds }} timeoutSeconds: {{ .Values.probes.startup.timeoutSeconds }} failureThreshold: {{ .Values.probes.startup.failureThreshold }} livenessProbe: - tcpSocket: - port: grpc + httpGet: + path: /healthz + port: health initialDelaySeconds: {{ .Values.probes.liveness.initialDelaySeconds }} periodSeconds: {{ .Values.probes.liveness.periodSeconds }} timeoutSeconds: {{ .Values.probes.liveness.timeoutSeconds }} failureThreshold: {{ .Values.probes.liveness.failureThreshold }} readinessProbe: - tcpSocket: - port: grpc + httpGet: + path: /readyz + port: health initialDelaySeconds: {{ .Values.probes.readiness.initialDelaySeconds }} periodSeconds: {{ .Values.probes.readiness.periodSeconds }} timeoutSeconds: {{ .Values.probes.readiness.timeoutSeconds }} @@ -142,7 +191,14 @@ spec: secretName: {{ .Values.server.tls.certSecretName }} - name: tls-client-ca secret: + {{- if or .Values.pkiInitJob.enabled (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} + secretName: {{ .Values.server.tls.certSecretName }} + items: + - key: ca.crt + path: ca.crt + {{- else }} secretName: {{ .Values.server.tls.clientCaSecretName }} + {{- end }} {{- end }} {{- with .Values.nodeSelector }} nodeSelector: diff --git a/deploy/helm/openshell/values-cert-manager.yaml b/deploy/helm/openshell/values-cert-manager.yaml new file mode 100644 index 0000000000..bb024d716f --- /dev/null +++ b/deploy/helm/openshell/values-cert-manager.yaml @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Merge after values.yaml when cert-manager CRDs are installed, e.g.: +# helm install ... -f values.yaml -f values-cert-manager.yaml +# Or add this file to skaffold manifests.helm.releases[].valuesFiles. +server: + disableTls: false + +pkiInitJob: + enabled: false + +certManager: + enabled: true diff --git a/deploy/helm/openshell/values-gateway.yaml b/deploy/helm/openshell/values-gateway.yaml new file mode 100644 index 0000000000..c43a4cd456 --- /dev/null +++ b/deploy/helm/openshell/values-gateway.yaml @@ -0,0 +1,26 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Gateway API overlay — enables a Gateway and GRPCRoute for external access. +# +# Requires Envoy Gateway in the cluster (installed via skaffold.yaml). +# Add this file to the openshell release valuesFiles to activate: +# uncomment values-gateway.yaml in deploy/helm/openshell/skaffold.yaml +# +# Envoy Gateway will create an Envoy proxy Deployment and a LoadBalancer +# Service (named envoy---*) in the openshell namespace. +# +# To reach the gateway from outside a k3d cluster, port-forward to that service: +# kubectl -n openshell get svc -l gateway.envoyproxy.io/owning-gateway-name=openshell +# kubectl -n openshell port-forward svc/ 8080:80 +# # then: grpcurl -plaintext localhost:8080 ... + +grpcRoute: + enabled: true + gateway: + create: true + className: "eg" + # Set one or more hostnames to scope the route, e.g.: + # hostnames: + # - openshell.example.com + hostnames: [] diff --git a/deploy/helm/openshell/values-keycloak.yaml b/deploy/helm/openshell/values-keycloak.yaml new file mode 100644 index 0000000000..42bb2ad4e6 --- /dev/null +++ b/deploy/helm/openshell/values-keycloak.yaml @@ -0,0 +1,36 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# OIDC configuration overlay for a local Keycloak instance in k3s. +# +# Run the one-time setup task first: +# mise run keycloak:k8s:setup +# +# Then layer this file on top of values.yaml when deploying: +# helm upgrade --install openshell . \ +# -f values.yaml -f values-skaffold.yaml -f values-keycloak.yaml +# +# Or add this file to skaffold.yaml valuesFiles for iterative dev. +# +# Issuer note: the setup task configures Keycloak with KC_HOSTNAME set to the +# in-cluster service hostname, so tokens always carry that hostname as `iss` +# regardless of how they were obtained (e.g. via a localhost port-forward). +# The gateway fetches JWKS from this URL inside the cluster. +# +# CLI token acquisition: keep a port-forward running while using openshell login: +# kubectl -n keycloak port-forward svc/keycloak 9090:80 + +server: + oidc: + # Must match KC_HOSTNAME set by keycloak:k8s:setup (in-cluster service hostname). + issuer: "http://keycloak.keycloak.svc.cluster.local/realms/openshell" + # Must match the client ID in the imported realm (openshell-cli). + audience: "openshell-cli" + # Short TTL for dev so JWKS key rotation is picked up quickly. + # Use 3600 (default) in production. + jwksTtl: 60 + # Keycloak puts realm roles at realm_access.roles in the JWT. + rolesClaim: "realm_access.roles" + # Leave both empty for authentication-only mode (any valid token is accepted). + adminRole: "openshell-admin" + userRole: "openshell-user" diff --git a/deploy/helm/openshell/values-skaffold.yaml b/deploy/helm/openshell/values-skaffold.yaml new file mode 100644 index 0000000000..24b60e1c62 --- /dev/null +++ b/deploy/helm/openshell/values-skaffold.yaml @@ -0,0 +1,12 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Merge with values.yaml for Skaffold-driven local image builds (see skaffold.yaml). +server: + sandboxImagePullPolicy: IfNotPresent + # Comment out to enforce mTLS (uses PKI secrets generated by pkiInitJob). + disableTls: true + +supervisor: + image: + pullPolicy: IfNotPresent diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index d698e8120d..ca30fa3713 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -7,8 +7,17 @@ replicaCount: 1 image: repository: ghcr.io/nvidia/openshell/gateway - pullPolicy: Always - tag: "latest" + pullPolicy: IfNotPresent + tag: "" + +# Supervisor image — provides the openshell-sandbox binary that is copied into +# sandbox pods via an init container. tag defaults to appVersion (same as the +# gateway image) so both stay in sync when the chart is released. +supervisor: + image: + repository: ghcr.io/nvidia/openshell/supervisor + pullPolicy: "" + tag: "" imagePullSecrets: [] nameOverride: "" @@ -34,9 +43,10 @@ securityContext: - ALL service: - type: NodePort + type: ClusterIP port: 8080 - nodePort: 30051 + healthPort: 8081 + metricsPort: 9090 # Pod restart behavior and health probe tuning. podLifecycle: @@ -87,7 +97,7 @@ server: # directly and requires client certificates. # Name of the Kubernetes Secret holding the NSSH1 HMAC handshake key. # The secret must contain a `secret` key with the hex-encoded HMAC key. - # For cluster deployments this is auto-created by the bootstrap process. + # By default a pre-install/pre-upgrade hook creates it when missing (see sshHandshake). sshHandshakeSecretName: "openshell-ssh-handshake" # Host gateway IP for sandbox pod hostAliases. When set, sandbox pods get # hostAliases entries mapping host.docker.internal and host.openshell.internal @@ -108,7 +118,113 @@ server: clientCaSecretName: openshell-server-client-ca # K8s secret mounted into sandbox pods for mTLS to the server clientTlsSecretName: openshell-client-tls + # OIDC (OpenID Connect) configuration for JWT-based authentication. + # When issuer is set, the server validates Bearer tokens on gRPC requests. + oidc: + # OIDC issuer URL (e.g. https://keycloak.example.com/realms/openshell). + issuer: "" + # Expected audience claim for the API resource server. + # This should match the server's --oidc-audience, NOT the CLI client ID. + audience: "openshell-cli" + # JWKS key cache TTL in seconds. + jwksTtl: 3600 + # Dot-separated path to the roles array in the JWT claims. + # Keycloak: "realm_access.roles", Entra ID: "roles", Okta: "groups". + rolesClaim: "" + # Role name for admin access. Leave empty (with userRole also empty) for + # authentication-only mode. Both must be set or both empty. + adminRole: "" + # Role name for standard user access. + userRole: "" + # Dot-separated path to the scopes array in the JWT claims. + scopesClaim: "" # NetworkPolicy restricting SSH ingress on sandbox pods to the gateway only. networkPolicy: enabled: true + +# NSSH1 SSH gateway handshake Secret (`server.sshHandshakeSecretName`). +# Helm hook creates it only when the Secret does not already exist (safe upgrades). +# Set sshHandshake.value from a gitignored values file for a stable dev secret. +sshHandshake: + hook: + enabled: true + # 64 hex chars (32 bytes), matching openshell-bootstrap. If empty, Helm generates + # a random value at install template time (two UUIDs, dashes stripped). + value: "" + +# PKI bootstrap via a pre-install/pre-upgrade hook Job. +# Generates a self-signed CA, server TLS secret, and client TLS secret using +# openssl (ECDSA P-256) inside the cluster. Key material is written directly to +# K8s Secrets and never appears in Helm release history. Idempotent: existing +# secrets are left untouched on upgrade. +# Air-gapped environments should override pkiInitJob.image with an image that has +# openssl and curl pre-installed (the default alpine image fetches them at runtime). +pkiInitJob: + enabled: true + image: + repository: alpine + tag: "3" + pullPolicy: IfNotPresent + # Days until the CA certificate expires. + caValidityDays: 3650 + # Days until server and client certificates expire. + certValidityDays: 3650 + # DNS SANs for the server certificate. + serverDnsNames: + - openshell + - openshell.openshell.svc + - openshell.openshell.svc.cluster.local + - localhost + - host.docker.internal + # IP SANs for the server certificate. + serverIpAddresses: + - 127.0.0.1 + +# cert-manager Certificate/Issuer resources (requires cert-manager CRDs in-cluster). +# Uses namespaced Issuers only (no ClusterIssuer). Does not install cert-manager itself. +certManager: + enabled: false + # Secret created for the intermediate CA (Certificate with isCA: true). + caSecretName: openshell-ca-tls + # Mount gateway client CA from the server TLS secret's ca.crt (populated by + # cert-manager for certs issued by a CA Issuer). Avoids a separate + # openshell-server-client-ca Secret. + clientCaFromServerTlsSecret: true + certificateDuration: 8760h + certificateRenewBefore: 720h + serverDnsNames: + - openshell + - openshell.openshell.svc + - openshell.openshell.svc.cluster.local + - localhost + - host.docker.internal + serverIpAddresses: + - 127.0.0.1 + +# Kubernetes Gateway API — HTTPRoute and Gateway resources. +# Requires a Gateway API controller in the cluster. Install Envoy Gateway via +# the skaffold.yaml releases or independently: +# helm install eg oci://docker.io/envoyproxy/gateway-helm \ +# --version v1.4.1 -n envoy-gateway-system --create-namespace +grpcRoute: + enabled: false + # Hostnames the GRPCRoute matches on. Leave empty to match all hosts. + hostnames: [] + gateway: + # When true, a Gateway resource is created in the release namespace. + # Set to false and provide name/namespace to attach to a pre-existing Gateway. + create: false + # GatewayClass to reference. Envoy Gateway installs one named "eg". + className: "eg" + # Name of the Gateway resource. Defaults to the chart fullname. + name: "" + # Namespace of the Gateway referenced by the GRPCRoute parentRef. + # Defaults to the release namespace. + namespace: "" + # Listener settings (only used when gateway.create is true). + listener: + port: 80 + protocol: HTTP + # "Same" restricts attached routes to the release namespace; "All" allows any namespace. + allowedRoutes: Same diff --git a/deploy/kube/gpu-manifests/nvidia-device-plugin-helmchart.yaml b/deploy/kube/gpu-manifests/nvidia-device-plugin-helmchart.yaml index 1cb0ca70ac..b5cd3fd2de 100644 --- a/deploy/kube/gpu-manifests/nvidia-device-plugin-helmchart.yaml +++ b/deploy/kube/gpu-manifests/nvidia-device-plugin-helmchart.yaml @@ -27,7 +27,7 @@ metadata: spec: repo: https://nvidia.github.io/k8s-device-plugin chart: nvidia-device-plugin - version: "0.18.2" + version: "0.19.1" targetNamespace: nvidia-device-plugin createNamespace: true valuesContent: |- diff --git a/deploy/kube/manifests/agent-sandbox.yaml b/deploy/kube/manifests/agent-sandbox.yaml index c7d671bce0..63d7c6845d 100644 --- a/deploy/kube/manifests/agent-sandbox.yaml +++ b/deploy/kube/manifests/agent-sandbox.yaml @@ -4111,4 +4111,10 @@ rules: - get - patch - update +- apiGroups: + - agents.x-k8s.io + resources: + - sandboxes/finalizers + verbs: + - update --- diff --git a/deploy/kube/manifests/envoy-gateway-openshell.yaml b/deploy/kube/manifests/envoy-gateway-openshell.yaml new file mode 100644 index 0000000000..583f2b41ba --- /dev/null +++ b/deploy/kube/manifests/envoy-gateway-openshell.yaml @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Envoy GatewayClass for OpenShell. +# +# Apply after a successful Skaffold deploy when gateway routing is enabled: +# mise run helm:gateway:apply + +--- +# The Envoy Gateway Helm chart does not create a GatewayClass — we manage it here. +# The proxy Service defaults to LoadBalancer, which klipper-lb handles in k3d. +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: eg +spec: + controllerName: gateway.envoyproxy.io/gatewayclass-controller diff --git a/deploy/kube/manifests/openshell-helmchart.yaml b/deploy/kube/manifests/openshell-helmchart.yaml index a09e0f3004..eba79364c1 100644 --- a/deploy/kube/manifests/openshell-helmchart.yaml +++ b/deploy/kube/manifests/openshell-helmchart.yaml @@ -31,6 +31,7 @@ spec: server: sandboxImage: ghcr.io/nvidia/openshell-community/sandboxes/base:latest sandboxImagePullPolicy: __SANDBOX_IMAGE_PULL_POLICY__ + supervisorImage: ghcr.io/nvidia/openshell/supervisor:latest dbUrl: __DB_URL__ sshGatewayHost: __SSH_GATEWAY_HOST__ sshGatewayPort: __SSH_GATEWAY_PORT__ @@ -38,6 +39,14 @@ spec: hostGatewayIP: __HOST_GATEWAY_IP__ disableGatewayAuth: __DISABLE_GATEWAY_AUTH__ disableTls: __DISABLE_TLS__ + oidc: + issuer: "__OIDC_ISSUER__" + audience: "__OIDC_AUDIENCE__" + jwksTtl: 3600 + rolesClaim: "__OIDC_ROLES_CLAIM__" + adminRole: "__OIDC_ADMIN_ROLE__" + userRole: "__OIDC_USER_ROLE__" + scopesClaim: "__OIDC_SCOPES_CLAIM__" tls: certSecretName: openshell-server-tls clientCaSecretName: openshell-server-client-ca diff --git a/deploy/man/openshell-gateway.8.md b/deploy/man/openshell-gateway.8.md new file mode 100644 index 0000000000..5e3c4eef2f --- /dev/null +++ b/deploy/man/openshell-gateway.8.md @@ -0,0 +1,210 @@ +--- +title: OPENSHELL-GATEWAY +section: 8 +header: OpenShell Manual +footer: openshell-gateway +date: 2025 +--- + +# NAME + +openshell-gateway - OpenShell gateway server daemon + +# SYNOPSIS + +**openshell-gateway** \[*OPTIONS*\] + +# DESCRIPTION + +**openshell-gateway** is the control-plane server for OpenShell. It +manages sandbox lifecycle, stores provider credentials, delivers +network and filesystem policies to sandboxes, routes inference +requests, and provides the SSH tunnel endpoint for CLI-to-sandbox +connections. + +When installed via RPM, the gateway runs as a systemd user service +with the Podman compute driver. Sandboxes are rootless Podman +containers on the host. + +The gateway exposes a single port (default 8080) with multiplexed +gRPC and HTTP, secured by mutual TLS (mTLS) by default. + +# OPTIONS + +**--bind-address** *IP* +: IP address to bind all listeners to. Default: **127.0.0.1**. + Environment: **OPENSHELL_BIND_ADDRESS**. + +**--port** *PORT* +: Port for the gRPC/HTTP API. Default: **8080**. + Environment: **OPENSHELL_SERVER_PORT**. + +**--health-port** *PORT* +: Port for unauthenticated health endpoints (/healthz, /readyz). + Set to 0 to disable. Default: **0**. + Environment: **OPENSHELL_HEALTH_PORT**. + +**--metrics-port** *PORT* +: Port for Prometheus metrics (/metrics). Set to 0 to disable. + Default: **0**. Environment: **OPENSHELL_METRICS_PORT**. + +**--log-level** *LEVEL* +: Log level: trace, debug, info, warn, error. Default: **info**. + Environment: **OPENSHELL_LOG_LEVEL**. + +**--db-url** *URL* +: SQLite database URL for state persistence. Required. + Environment: **OPENSHELL_DB_URL**. + +**--drivers** *DRIVER*\[,*DRIVER*\] +: Compute driver. Accepts a comma-delimited list. The gateway + currently requires exactly one driver. Options: **podman**, + **docker**, **kubernetes**. Default: **kubernetes**. + Environment: **OPENSHELL_DRIVERS**. + +**--tls-cert** *PATH* +: Path to server TLS certificate file. Required unless + **--disable-tls** is set. Environment: **OPENSHELL_TLS_CERT**. + +**--tls-key** *PATH* +: Path to server TLS private key file. Required unless + **--disable-tls** is set. Environment: **OPENSHELL_TLS_KEY**. + +**--tls-client-ca** *PATH* +: Path to CA certificate for client certificate verification (mTLS). + Required unless **--disable-tls** is set. + Environment: **OPENSHELL_TLS_CLIENT_CA**. + +**--disable-tls** +: Disable TLS entirely and listen on plaintext HTTP. When the bind + address is **0.0.0.0** (the RPM default), disabling TLS exposes the + API to the entire network without authentication. Only use when the + gateway sits behind a TLS-terminating reverse proxy, or restrict + **--bind-address** to **127.0.0.1**. + Environment: **OPENSHELL_DISABLE_TLS**. + +**--disable-gateway-auth** +: Disable mTLS client certificate requirement. The TLS handshake + accepts connections without a client certificate. Ignored when + **--disable-tls** is set. + Environment: **OPENSHELL_DISABLE_GATEWAY_AUTH**. + +**--sandbox-image** *IMAGE* +: Default container image for sandboxes. + Environment: **OPENSHELL_SANDBOX_IMAGE**. + +**--sandbox-image-pull-policy** *POLICY* +: Image pull policy: Always, IfNotPresent, Never. + Environment: **OPENSHELL_SANDBOX_IMAGE_PULL_POLICY**. + +**--ssh-handshake-secret** *SECRET* +: Shared secret for gateway-to-sandbox SSH handshake. + Environment: **OPENSHELL_SSH_HANDSHAKE_SECRET**. + +**--ssh-handshake-skew-secs** *SECONDS* +: Allowed clock skew in seconds for SSH handshake. Default: **30**. + Environment: **OPENSHELL_SSH_HANDSHAKE_SKEW_SECS**. + +**--ssh-gateway-host** *HOST* +: Public host for the SSH gateway endpoint. Default: **127.0.0.1**. + Environment: **OPENSHELL_SSH_GATEWAY_HOST**. + +**--ssh-gateway-port** *PORT* +: Public port for the SSH gateway endpoint. Default: **8080**. + Environment: **OPENSHELL_SSH_GATEWAY_PORT**. + +**--grpc-endpoint** *URL* +: gRPC endpoint for sandbox callbacks. Should be reachable from + within sandbox containers. + Environment: **OPENSHELL_GRPC_ENDPOINT**. + +# SYSTEMD INTEGRATION + +The RPM installs a systemd user unit at +*/usr/lib/systemd/user/openshell-gateway.service*. Manage the gateway +with standard systemd commands: + + systemctl --user enable --now openshell-gateway + systemctl --user status openshell-gateway + systemctl --user restart openshell-gateway + systemctl --user stop openshell-gateway + +View logs: + + journalctl --user -u openshell-gateway + journalctl --user -u openshell-gateway -f + +The unit runs two **ExecStartPre** scripts on first start: + +1. **init-pki.sh** generates a self-signed PKI bundle for mTLS. +2. **init-gateway-env.sh** generates the environment configuration + file with an auto-generated SSH handshake secret. + +Both scripts are idempotent and skip generation if their output files +already exist. + +To persist the service across logouts: + + sudo loginctl enable-linger $USER + +# CONFIGURATION + +The systemd user unit reads configuration from +*~/.config/openshell/gateway.env*. See **openshell-gateway.env**(5) +for the full variable reference. + +To override individual settings without modifying gateway.env: + + systemctl --user edit openshell-gateway + +This creates a drop-in override that persists across package upgrades. + +# FILES + +*/usr/bin/openshell-gateway* +: Gateway binary. + +*/usr/lib/systemd/user/openshell-gateway.service* +: Systemd user unit file. + +*/usr/libexec/openshell/init-pki.sh* +: PKI bootstrap script. + +*/usr/libexec/openshell/init-gateway-env.sh* +: Gateway environment file generator. + +*~/.config/openshell/gateway.env* +: Gateway environment configuration (generated on first start). + +*~/.local/state/openshell/tls/* +: Auto-generated TLS certificates. + +*~/.local/state/openshell/gateway.db* +: SQLite database for gateway state. + +*~/.config/openshell/gateways/openshell/mtls/* +: Client mTLS certificates for CLI auto-discovery. + +# EXAMPLES + +Start the gateway as a systemd user service: + + systemctl --user enable --now openshell-gateway + +Check gateway health from the CLI: + + openshell gateway add --local https://127.0.0.1:8080 + openshell status + +Override the API port via a systemd drop-in: + + systemctl --user edit openshell-gateway + # Add: [Service] + # Add: Environment=OPENSHELL_SERVER_PORT=9090 + +# SEE ALSO + +**openshell**(1), **openshell-gateway.env**(5), **systemctl**(1), +**journalctl**(1), **loginctl**(1), **podman**(1) + +Full documentation: *https://docs.nvidia.com/openshell/* diff --git a/deploy/man/openshell-gateway.env.5.md b/deploy/man/openshell-gateway.env.5.md new file mode 100644 index 0000000000..a4e715edd4 --- /dev/null +++ b/deploy/man/openshell-gateway.env.5.md @@ -0,0 +1,168 @@ +--- +title: OPENSHELL-GATEWAY.ENV +section: 5 +header: OpenShell Manual +footer: openshell-gateway +date: 2025 +--- + +# NAME + +openshell-gateway.env - OpenShell gateway environment configuration + +# DESCRIPTION + +The **openshell-gateway.env** file contains environment variables that +configure the OpenShell gateway server when running as a systemd user +service. It is generated automatically on first start by +**init-gateway-env.sh** and is not overwritten on subsequent starts or +package upgrades. + +The file uses the standard systemd **EnvironmentFile** format: one +**KEY=VALUE** pair per line. Lines beginning with **#** are comments. +Shell variable expansion is not performed. + +# LOCATION + +The file is located at: + + ~/.config/openshell/gateway.env + +The systemd user unit reads it via: + + EnvironmentFile=-~/.config/openshell/gateway.env + +The **-** prefix means the service starts normally if the file does not +exist (the unit has built-in defaults for all required settings except +the SSH handshake secret). + +# VARIABLES + +## Required + +**OPENSHELL_SSH_HANDSHAKE_SECRET** +: Shared HMAC secret for gateway-to-sandbox SSH handshake + authentication. Auto-generated as a 32-byte hex string on first + start. To regenerate: **openssl rand -hex 32**. + +## Gateway + +**OPENSHELL_BIND_ADDRESS** (default: 0.0.0.0) +: IP address to bind all listeners to. The RPM default of **0.0.0.0** + exposes the gateway on all network interfaces; mTLS must remain + enabled to prevent unauthenticated access. Set to **127.0.0.1** for + local-only access. + +**OPENSHELL_SERVER_PORT** (default: 8080) +: Port for the multiplexed gRPC/HTTP API. + +**OPENSHELL_HEALTH_PORT** (default: 0) +: Port for unauthenticated health endpoints (/healthz, /readyz). + Set to a non-zero value to enable a dedicated health listener. + +**OPENSHELL_METRICS_PORT** (default: 0) +: Port for Prometheus metrics endpoint (/metrics). Set to a + non-zero value to enable a dedicated metrics listener. + +**OPENSHELL_LOG_LEVEL** (default: info) +: Log verbosity: **trace**, **debug**, **info**, **warn**, **error**. + +**OPENSHELL_DRIVERS** (default: podman) +: Compute driver for sandbox management. Options: **podman**, + **docker**, **kubernetes**. The RPM unit defaults to **podman**. + +**OPENSHELL_DB_URL** (default: sqlite://$XDG_STATE_HOME/openshell/gateway.db) +: SQLite database URL for gateway state persistence. + +**OPENSHELL_DISABLE_GATEWAY_AUTH** (default: unset) +: Set to **true** to disable mTLS client certificate verification. + +## TLS + +**OPENSHELL_TLS_CERT** (default: auto-generated path) +: Path to server TLS certificate. + +**OPENSHELL_TLS_KEY** (default: auto-generated path) +: Path to server TLS private key. + +**OPENSHELL_TLS_CLIENT_CA** (default: auto-generated path) +: Path to CA certificate for client certificate verification. + +**OPENSHELL_DISABLE_TLS** (default: unset) +: Set to **true** to disable TLS entirely and listen on plaintext + HTTP. Not recommended for production. When the bind address is + **0.0.0.0** (the RPM default), disabling TLS exposes the API to the + entire network without authentication. Restrict + **OPENSHELL_BIND_ADDRESS** to **127.0.0.1** or place the gateway + behind a TLS-terminating reverse proxy. + +**OPENSHELL_PODMAN_TLS_CA** (default: auto-generated path) +: CA certificate bind-mounted into sandbox containers. + +**OPENSHELL_PODMAN_TLS_CERT** (default: auto-generated path) +: Client certificate bind-mounted into sandbox containers. + +**OPENSHELL_PODMAN_TLS_KEY** (default: auto-generated path) +: Client private key bind-mounted into sandbox containers. + +## Images + +**OPENSHELL_SUPERVISOR_IMAGE** (default: ghcr.io/nvidia/openshell/supervisor:latest) +: OCI image containing the supervisor binary, mounted read-only + into sandbox containers. + +**OPENSHELL_SANDBOX_IMAGE** (default: ghcr.io/nvidia/openshell-community/sandboxes/base:latest) +: Default OCI image for sandbox containers. + +**OPENSHELL_SANDBOX_IMAGE_PULL_POLICY** (default: missing) +: When to pull sandbox images: **always** (every sandbox creation), + **missing** (only if not cached locally), **never** (use cached + only), **newer** (pull if a newer version exists). + +## Podman Driver + +**OPENSHELL_PODMAN_SOCKET** (default: $XDG_RUNTIME_DIR/podman/podman.sock) +: Path to the Podman API Unix socket. + +**OPENSHELL_NETWORK_NAME** (default: openshell) +: Name of the Podman bridge network for sandbox containers. Created + automatically if it does not exist. + +**OPENSHELL_STOP_TIMEOUT** (default: 10) +: Seconds to wait after SIGTERM before sending SIGKILL when stopping + a sandbox container. + +# EXAMPLES + +Change the API port to 9090: + + OPENSHELL_SERVER_PORT=9090 + +Pin sandbox images to a specific version: + + OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/nvidia/openshell/supervisor:v0.0.37 + OPENSHELL_SANDBOX_IMAGE=ghcr.io/nvidia/openshell-community/sandboxes/base:v0.0.37 + +Air-gapped deployment (pre-loaded images, no registry access): + + OPENSHELL_SANDBOX_IMAGE_PULL_POLICY=never + +Enable debug logging: + + OPENSHELL_LOG_LEVEL=debug + +Use externally-managed TLS certificates: + + OPENSHELL_TLS_CERT=/etc/pki/tls/certs/openshell.crt + OPENSHELL_TLS_KEY=/etc/pki/tls/private/openshell.key + OPENSHELL_TLS_CLIENT_CA=/etc/pki/tls/certs/openshell-ca.crt + +Disable TLS (behind a reverse proxy): + + OPENSHELL_DISABLE_TLS=true + +# SEE ALSO + +**openshell-gateway**(8), **openshell**(1), **systemd.exec**(5) + +Full documentation: *https://docs.nvidia.com/openshell/* diff --git a/deploy/man/openshell.1.md b/deploy/man/openshell.1.md new file mode 100644 index 0000000000..65e060899b --- /dev/null +++ b/deploy/man/openshell.1.md @@ -0,0 +1,217 @@ +--- +title: OPENSHELL +section: 1 +header: OpenShell Manual +footer: openshell +date: 2025 +--- + +# NAME + +openshell - CLI for managing OpenShell sandboxes, gateways, and providers + +# SYNOPSIS + +**openshell** \[*OPTIONS*\] *COMMAND* \[*ARGS*\] + +# DESCRIPTION + +**openshell** is the command-line interface for OpenShell, a platform +providing safe, sandboxed runtimes for autonomous AI agents. It manages +the gateway control plane, sandbox lifecycle, credential providers, +network policies, and inference routing. + +The CLI communicates with a gateway server over gRPC. The gateway can +run as a systemd user service (RPM deployment with Podman driver), a +Docker container with embedded K3s, or behind a cloud reverse proxy. + +# COMMANDS + +## Gateway Management + +**gateway start** +: Deploy a new gateway using Docker (not applicable to RPM deployments; + use **systemctl --user start openshell-gateway** instead). + +**gateway stop** +: Stop a Docker-managed gateway (use **systemctl --user stop + openshell-gateway** for RPM deployments). + +**gateway destroy** \[**--name** *NAME*\] +: Destroy a gateway. For RPM deployments, this removes the CLI + registration only. + +**gateway add** *ENDPOINT* \[**--local**\] \[**--name** *NAME*\] \[**--remote** *USER@HOST*\] +: Register an existing gateway with the CLI. + +**gateway select** \[*NAME*\] +: List registered gateways or switch the active gateway. + +**gateway info** \[**--name** *NAME*\] +: Show deployment details for a gateway. + +**gateway login** +: Re-authenticate with a cloud gateway. + +**status** +: Check the health of the active gateway. + +## Sandbox Management + +**sandbox create** \[**--from** *IMAGE*\] \[**--policy** *FILE*\] \[**--provider** *NAME*\] \[**--gpu**\] \[**--upload** *SRC:DST*\] \[**--forward** *PORT*\] \[**--** *COMMAND*\] +: Create a new sandbox. If no gateway exists, auto-bootstraps one + (Docker mode only). + +**sandbox list** \[**--selector** *LABEL*\] +: List all sandboxes on the active gateway. + +**sandbox get** *NAME* +: Show details for a sandbox. + +**sandbox delete** *NAME* \| **--all** +: Delete one or all sandboxes. + +**sandbox connect** *NAME* \[**--editor** *EDITOR*\] +: SSH into a running sandbox. + +**sandbox exec** **-n** *NAME* \[**--workdir** *DIR*\] **--** *COMMAND* +: Execute a command in a sandbox. + +**sandbox upload** *NAME* *LOCAL* *REMOTE* +: Upload files to a sandbox. + +**sandbox download** *NAME* *REMOTE* *LOCAL* +: Download files from a sandbox. + +## Policy Management + +**policy get** *SANDBOX* \[**--full**\] +: Show the active policy for a sandbox. + +**policy set** *SANDBOX* **--policy** *FILE* \[**--wait**\] +: Apply a policy to a sandbox. + +**policy update** *SANDBOX* \[**--add-endpoint** *SPEC*\] \[**--add-allow** *RULE*\] +: Incrementally update a sandbox policy. + +**policy list** *SANDBOX* +: Show policy revision history. + +**policy prove** **--policy** *FILE* \[**--credentials** *FILE*\] +: Verify policy properties. + +## Provider Management + +**provider create** **--name** *NAME* **--type** *TYPE* \[**--from-existing**\] \[**--credential** *KEY=VALUE*\] +: Create a credential provider. + +**provider list** +: List all providers. + +**provider get** *NAME* +: Show provider details. + +**provider update** *NAME* \[**--from-existing**\] \[**--credential** *KEY=VALUE*\] +: Update provider credentials. + +**provider delete** *NAME* +: Delete a provider. + +## Inference Routing + +**inference set** **--provider** *NAME* **--model** *MODEL* +: Configure inference routing. + +**inference get** +: Show current inference configuration. + +**inference update** \[**--model** *MODEL*\] +: Update inference configuration. + +## Other + +**logs** *SANDBOX* \[**--tail**\] +: View sandbox logs. + +**forward start** *PORT* *SANDBOX* \[**-d**\] +: Start port forwarding to a sandbox. + +**forward stop** *PORT* +: Stop port forwarding. + +**forward list** +: List active port forwards. + +**term** +: Open the real-time TUI dashboard. + +**doctor check** \| **logs** \| **exec** \| **llm.txt** +: Diagnostic tools (Docker/K3s mode only; see **TROUBLESHOOTING** + section for RPM alternatives). + +**completions** *SHELL* +: Generate shell completions (bash, zsh, fish). + +# GLOBAL OPTIONS + +**-g**, **--gateway** *NAME* +: Target a specific gateway by name. + +**--gateway-endpoint** *URL* +: Connect to a gateway by URL directly. + +**-h**, **--help** +: Print help information. + +**-V**, **--version** +: Print version. + +# ENVIRONMENT + +**OPENSHELL_GATEWAY** +: Default gateway name (overrides active gateway). + +**OPENSHELL_GATEWAY_ENDPOINT** +: Direct gateway URL (bypasses metadata lookup). + +**ANTHROPIC_API_KEY**, **OPENAI_API_KEY**, **OPENROUTER_API_KEY** +: API keys discovered by auto-provider creation. + +**GITHUB_TOKEN**, **GH_TOKEN** +: GitHub credentials for provider auto-discovery. + +# FILES + +*~/.config/openshell/gateways/* +: Gateway metadata and mTLS certificates. + +*~/.config/openshell/active_gateway* +: Name of the currently active gateway. + +# EXAMPLES + +Register the local RPM gateway and create a sandbox: + + openshell gateway add --local https://127.0.0.1:8080 + openshell sandbox create -- claude + +List sandboxes and connect to one: + + openshell sandbox list + openshell sandbox connect my-sandbox + +Create a provider from a local environment variable: + + openshell provider create --name openai --type openai --from-existing + +Check gateway health: + + openshell status + +# SEE ALSO + +**openshell-gateway**(8), **openshell-gateway.env**(5) + +Full documentation: *https://docs.nvidia.com/openshell/* + +Run **openshell** *COMMAND* **--help** for detailed help on any command. diff --git a/deploy/rpm/CONFIGURATION.md b/deploy/rpm/CONFIGURATION.md new file mode 100644 index 0000000000..04caaaae57 --- /dev/null +++ b/deploy/rpm/CONFIGURATION.md @@ -0,0 +1,253 @@ +# OpenShell Gateway Configuration (RPM) + +Configuration reference for the OpenShell gateway when installed via +the RPM package on Fedora and RHEL systems. + +For first-time setup, see QUICKSTART.md. For troubleshooting, see +TROUBLESHOOTING.md. + +## TLS (mTLS) + +The RPM enables mutual TLS by default. The gateway requires a valid +client certificate for all API connections, protecting the API even +though it listens on all interfaces (`0.0.0.0`). + +### Auto-generated certificates + +On first start, the `init-pki.sh` script generates certificates using +OpenSSL: + +| File | Purpose | Location | +|------|---------|----------| +| CA certificate | Root of trust | `~/.local/state/openshell/tls/ca.crt` | +| CA private key | Signs server and client certs | `~/.local/state/openshell/tls/ca.key` | +| Server certificate | Gateway TLS identity | `~/.local/state/openshell/tls/server/tls.crt` | +| Server private key | Gateway TLS key | `~/.local/state/openshell/tls/server/tls.key` | +| Client certificate | CLI and sandbox identity | `~/.local/state/openshell/tls/client/tls.crt` | +| Client private key | CLI and sandbox key | `~/.local/state/openshell/tls/client/tls.key` | + +Client certificates are also copied to the CLI auto-discovery directory: + +``` +~/.config/openshell/gateways/openshell/mtls/ + ca.crt + tls.crt + tls.key +``` + +The CLI automatically discovers these certificates when connecting to a +gateway on `localhost` or `127.0.0.1`. + +### Server certificate SANs + +The auto-generated server certificate includes these Subject Alternative +Names: + +- `localhost` +- `openshell` +- `openshell.openshell.svc` +- `openshell.openshell.svc.cluster.local` +- `host.containers.internal` +- `host.docker.internal` +- `127.0.0.1` + +To connect from a remote machine, you need externally-managed +certificates with additional SANs. See "Remote CLI access" in +TROUBLESHOOTING.md. + +### Using externally-managed certificates + +To use certificates from an external CA or cert-manager: + +1. Place the server cert, key, and CA cert on the filesystem. + +1. Edit `~/.config/openshell/gateway.env` or use + `systemctl --user edit openshell-gateway` to override: + + ```shell + OPENSHELL_TLS_CERT=/path/to/server/tls.crt + OPENSHELL_TLS_KEY=/path/to/server/tls.key + OPENSHELL_TLS_CLIENT_CA=/path/to/ca.crt + ``` + +1. Place the client cert where the CLI expects it: + + ``` + ~/.config/openshell/gateways/openshell/mtls/ + ca.crt + tls.crt + tls.key + ``` + +### Rotating certificates + +Delete the TLS state directory and restart the gateway: + +```shell +rm -rf ~/.local/state/openshell/tls +systemctl --user restart openshell-gateway +``` + +The gateway regenerates the PKI on next start. + +### Disabling TLS + +> **WARNING:** The RPM gateway binds to all interfaces (`0.0.0.0`) by +> default. With TLS disabled, the gateway API is exposed to the entire +> network with **no authentication**. Any host that can reach the +> gateway port has full access, including the ability to create +> sandboxes, execute arbitrary code, and access configured credentials. +> Only disable TLS when the gateway is behind a TLS-terminating reverse +> proxy that enforces its own authentication. When disabling TLS without +> a reverse proxy, restrict `OPENSHELL_BIND_ADDRESS` to `127.0.0.1`. + +To disable TLS (not recommended for production): + +1. Edit `~/.config/openshell/gateway.env`: + + ```shell + OPENSHELL_DISABLE_TLS=true + ``` + +1. Comment out the `OPENSHELL_TLS_*` and `OPENSHELL_PODMAN_TLS_*` + variables if they are set. + +1. Restart the gateway. + +## Sandbox TLS + +When mTLS is enabled, the Podman driver bind-mounts the client +certificates into each sandbox container so the supervisor process can +establish an mTLS connection back to the gateway. + +The following environment variables control the host-side paths of the +client certificates that are mounted into sandbox containers: + +| Variable | Description | +|----------|-------------| +| `OPENSHELL_PODMAN_TLS_CA` | CA certificate (host path) | +| `OPENSHELL_PODMAN_TLS_CERT` | Client certificate (host path) | +| `OPENSHELL_PODMAN_TLS_KEY` | Client private key (host path) | + +Inside the container, the supervisor reads them from: + +- `/etc/openshell/tls/client/ca.crt` +- `/etc/openshell/tls/client/tls.crt` +- `/etc/openshell/tls/client/tls.key` + +On SELinux-enabled systems, the Podman driver automatically applies the +`:z` relabel option to these bind mounts. No manual SELinux +configuration is required. + +## Configuration reference + +All settings are controlled via environment variables. The user unit +reads from `~/.config/openshell/gateway.env` (generated on first start) +and from `Environment=` directives in the systemd unit. + +Values in `gateway.env` override the unit defaults. Use +`systemctl --user edit openshell-gateway` to add overrides that persist +across package upgrades. + +### Gateway settings + +| Variable | Default | Description | +|----------|---------|-------------| +| `OPENSHELL_BIND_ADDRESS` | `0.0.0.0` | IP address to bind all listeners to. The default exposes the gateway on all interfaces; mTLS must remain enabled to prevent unauthenticated access. Set to `127.0.0.1` for local-only access. | +| `OPENSHELL_SERVER_PORT` | `8080` | Port for the gRPC/HTTP API | +| `OPENSHELL_HEALTH_PORT` | `0` (disabled) | Port for unauthenticated health endpoints (`/healthz`, `/readyz`). Set to a non-zero value to enable. | +| `OPENSHELL_METRICS_PORT` | `0` (disabled) | Port for Prometheus metrics (`/metrics`). Set to a non-zero value to enable. | +| `OPENSHELL_LOG_LEVEL` | `info` | Log level: `trace`, `debug`, `info`, `warn`, `error` | +| `OPENSHELL_DRIVERS` | `podman` | Compute driver (`podman`, `docker`, `kubernetes`) | +| `OPENSHELL_DB_URL` | `sqlite://$XDG_STATE_HOME/openshell/gateway.db` | SQLite database URL for state persistence | +| `OPENSHELL_SSH_HANDSHAKE_SECRET` | (auto-generated) | Shared secret for sandbox SSH authentication | +| `OPENSHELL_DISABLE_GATEWAY_AUTH` | (unset) | Set to `true` to skip mTLS client certificate checks | + +### TLS settings + +| Variable | Default | Description | +|----------|---------|-------------| +| `OPENSHELL_TLS_CERT` | (auto-generated path) | Server TLS certificate | +| `OPENSHELL_TLS_KEY` | (auto-generated path) | Server TLS private key | +| `OPENSHELL_TLS_CLIENT_CA` | (auto-generated path) | CA for client certificate verification | +| `OPENSHELL_DISABLE_TLS` | (unset) | Set to `true` to disable TLS | +| `OPENSHELL_PODMAN_TLS_CA` | (auto-generated path) | CA cert mounted into sandbox containers | +| `OPENSHELL_PODMAN_TLS_CERT` | (auto-generated path) | Client cert mounted into sandbox containers | +| `OPENSHELL_PODMAN_TLS_KEY` | (auto-generated path) | Client key mounted into sandbox containers | + +### Sandbox settings + +| Variable | Default | Description | +|----------|---------|-------------| +| `OPENSHELL_SUPERVISOR_IMAGE` | `ghcr.io/nvidia/openshell/supervisor:latest` | Supervisor binary OCI image | +| `OPENSHELL_SANDBOX_IMAGE` | `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` | Default sandbox base image | +| `OPENSHELL_SANDBOX_IMAGE_PULL_POLICY` | `missing` | Image pull policy: `always`, `missing`, `never`, `newer` | + +### Podman driver settings + +| Variable | Default | Description | +|----------|---------|-------------| +| `OPENSHELL_PODMAN_SOCKET` | `$XDG_RUNTIME_DIR/podman/podman.sock` | Podman API Unix socket path | +| `OPENSHELL_NETWORK_NAME` | `openshell` | Podman bridge network name for sandbox containers | +| `OPENSHELL_STOP_TIMEOUT` | `10` | Container stop timeout in seconds (SIGTERM then SIGKILL) | + +### Image management + +The gateway pulls container images automatically on first sandbox +creation. The default pull policy is `missing`, which means images are +pulled once and then cached by Podman. + +To update cached images: + +```shell +podman pull ghcr.io/nvidia/openshell/supervisor:latest +podman pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest +``` + +Or set `OPENSHELL_SANDBOX_IMAGE_PULL_POLICY=always` to pull on every +sandbox creation. + +To pin specific image versions instead of `:latest`: + +```shell +OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/nvidia/openshell/supervisor:v0.0.37 +OPENSHELL_SANDBOX_IMAGE=ghcr.io/nvidia/openshell-community/sandboxes/base:v0.0.37 +``` + +For air-gapped environments: + +1. On a connected machine, pull and save the images: + + ```shell + podman pull ghcr.io/nvidia/openshell/supervisor:latest + podman pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest + podman save -o supervisor.tar ghcr.io/nvidia/openshell/supervisor:latest + podman save -o sandbox.tar ghcr.io/nvidia/openshell-community/sandboxes/base:latest + ``` + +1. Transfer the tarballs to the air-gapped host and load them: + + ```shell + podman load -i supervisor.tar + podman load -i sandbox.tar + ``` + +1. Set pull policy to `never`: + + ```shell + OPENSHELL_SANDBOX_IMAGE_PULL_POLICY=never + ``` + +## File locations + +| Purpose | Path | +|---------|------| +| Gateway binary | `/usr/bin/openshell-gateway` | +| CLI binary | `/usr/bin/openshell` | +| Systemd user unit | `/usr/lib/systemd/user/openshell-gateway.service` | +| PKI bootstrap script | `/usr/libexec/openshell/init-pki.sh` | +| Env generator script | `/usr/libexec/openshell/init-gateway-env.sh` | +| TLS certificates | `~/.local/state/openshell/tls/` | +| CLI client certs | `~/.config/openshell/gateways/openshell/mtls/` | +| Gateway database | `~/.local/state/openshell/gateway.db` | +| Gateway configuration | `~/.config/openshell/gateway.env` | diff --git a/deploy/rpm/QUICKSTART.md b/deploy/rpm/QUICKSTART.md new file mode 100644 index 0000000000..b25be7d776 --- /dev/null +++ b/deploy/rpm/QUICKSTART.md @@ -0,0 +1,164 @@ +# OpenShell RPM Quick Start + +Get from `dnf install` to a running sandbox in five minutes. + +## Prerequisites + +### Podman (rootless) + +The gateway uses rootless Podman for sandbox containers. Verify +Podman is installed and the cgroup version is v2: + +```shell +podman --version +podman info --format '{{.Host.CgroupsVersion}}' +``` + +The cgroup version must be `v2`. If it reports `v1`, enable the +unified cgroup hierarchy and reboot: + +```shell +sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1" +sudo reboot +``` + +### Subordinate UID/GID ranges + +Rootless containers require subordinate UID/GID mappings: + +```shell +grep $USER /etc/subuid /etc/subgid +``` + +If empty, add entries: + +```shell +sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER +``` + +### Podman socket + +The gateway communicates with Podman over its API socket. Enable +socket activation: + +```shell +systemctl --user enable --now podman.socket +``` + +### Network access + +The gateway pulls container images from ghcr.io on first sandbox +creation. Ensure the host can reach ghcr.io over HTTPS (port 443). + +For air-gapped environments, pre-load images with `podman pull` and +set `OPENSHELL_SANDBOX_IMAGE_PULL_POLICY=never` in +`~/.config/openshell/gateway.env`. See CONFIGURATION.md for details. + +## Start the gateway + +```shell +systemctl --user enable --now openshell-gateway +``` + +On first start, the gateway automatically generates: + +- A self-signed PKI bundle (CA, server cert, client cert) for mTLS +- An SSH handshake secret for sandbox authentication +- A commented configuration file at `~/.config/openshell/gateway.env` + +> **Note:** The gateway binds to all interfaces (`0.0.0.0`) by default. +> Mutual TLS (mTLS) is enabled automatically on first start, requiring a +> valid client certificate for every connection. Do not disable TLS +> without restricting the bind address to `127.0.0.1`. See +> CONFIGURATION.md for details. + +Verify the service is running: + +```shell +systemctl --user status openshell-gateway +``` + +## Register the gateway with the CLI + +The CLI needs to know where the gateway is. Register it: + +```shell +openshell gateway add --local https://127.0.0.1:8080 +``` + +This discovers the pre-provisioned mTLS certificates at +`~/.config/openshell/gateways/openshell/mtls/` and sets the gateway +as active. + +Verify the connection: + +```shell +openshell status +``` + +## Persist across reboots + +By default, user services stop when you log out. To keep the gateway +running after logout and across reboots: + +```shell +sudo loginctl enable-linger $USER +``` + +Without this, the gateway and all running sandboxes are killed when +your login session ends. This is required for any headless or +production use. + +## Create your first sandbox + +Set your API key in the environment, then create a sandbox: + +```shell +export ANTHROPIC_API_KEY=sk-... +openshell sandbox create -- claude +``` + +The CLI detects the agent, prompts to create a credential provider +from your local environment, pulls the sandbox image from ghcr.io, +and connects you to the running sandbox. + +Other agents: + +```shell +openshell sandbox create -- opencode +openshell sandbox create -- codex +``` + +## Set up providers manually + +If you prefer to configure providers before creating sandboxes: + +```shell +# Create a provider from a local environment variable +openshell provider create --name anthropic --type anthropic --from-existing + +# Or supply the credential directly +openshell provider create --name openai --type openai \ + --credential OPENAI_API_KEY=sk-... + +# List configured providers +openshell provider list +``` + +## Configure inference routing (optional) + +To route inference requests through a specific provider and model: + +```shell +openshell inference set --provider openai --model gpt-4 +openshell inference get +``` + +## Next steps + +- See CONFIGURATION.md for TLS settings, environment variables, and + file locations. +- See TROUBLESHOOTING.md for CLI compatibility notes, remote access, + and common issues. +- Run `man openshell` for the CLI reference. +- Run `man openshell-gateway` for the gateway daemon reference. diff --git a/deploy/rpm/TROUBLESHOOTING.md b/deploy/rpm/TROUBLESHOOTING.md new file mode 100644 index 0000000000..78ef4c475c --- /dev/null +++ b/deploy/rpm/TROUBLESHOOTING.md @@ -0,0 +1,230 @@ +# OpenShell RPM Troubleshooting + +Troubleshooting guide, CLI compatibility notes, remote access setup, +and upgrade procedures for the RPM deployment. + +## CLI compatibility + +The RPM installs the gateway as a systemd user service with the Podman +compute driver. The published online docs and some CLI commands assume +a Docker/K3s deployment model. This section clarifies which commands +work, which do not, and what to use instead. + +### Commands that work normally + +All sandbox, provider, policy, inference, and settings commands +communicate with the gateway over gRPC and work identically regardless +of deployment mode: + +``` +openshell status +openshell sandbox create|list|get|delete|connect|exec +openshell logs +openshell provider create|list|get|update|delete +openshell policy get|set|update|list|prove +openshell inference set|get|update +openshell settings get|set +openshell forward start|stop|list +openshell term +openshell gateway add|select|info +openshell gateway destroy (removes CLI registration only) +``` + +### Commands that do not apply + +These commands manage Docker container lifecycle and are not applicable +to the RPM/systemd deployment. Use the systemd equivalents instead. + +| CLI command | RPM alternative | +|-------------|-----------------| +| `openshell gateway start` | `systemctl --user start openshell-gateway` | +| `openshell gateway stop` | `systemctl --user stop openshell-gateway` | +| `openshell doctor check` | `systemctl --user status openshell-gateway` | +| `openshell doctor logs` | `journalctl --user -u openshell-gateway` | +| `openshell doctor logs --tail` | `journalctl --user -u openshell-gateway -f` | +| `openshell doctor exec` | Not applicable (no K3s container) | + +### Building from local Dockerfiles + +`openshell sandbox create --from ./Dockerfile` builds via Docker and +pushes into K3s containerd. With the Podman driver, build the image +with Podman and reference it directly: + +```shell +podman build -t my-sandbox ./my-dir +openshell sandbox create --from localhost/my-sandbox +``` + +## Remote CLI access + +The auto-generated server certificate only includes SANs for +`localhost`, `127.0.0.1`, and Podman-internal names. To connect from a +different machine, choose one of the following approaches. + +### Option 1: SSH tunnel (simplest) + +Forward the gateway port over SSH and connect via localhost: + +```shell +# On the remote CLI machine: +ssh -L 8080:127.0.0.1:8080 user@gateway-host + +# In another terminal on the same machine: +# Copy the client certs from the gateway host first: +scp -r user@gateway-host:~/.config/openshell/gateways/openshell/mtls/ \ + ~/.config/openshell/gateways/openshell/mtls/ + +openshell gateway add --local https://127.0.0.1:8080 +openshell status +``` + +### Option 2: Externally-managed certificates + +Generate certificates that include the server's hostname or IP in the +SANs. See "Using externally-managed certificates" in CONFIGURATION.md. + +After placing the server and client certs, register from the remote +CLI: + +```shell +# Copy client certs to the remote CLI machine +mkdir -p ~/.config/openshell/gateways/openshell/mtls/ +cp ca.crt tls.crt tls.key ~/.config/openshell/gateways/openshell/mtls/ + +openshell gateway add --local https://:8080 +``` + +### Firewall + +For remote access, open the gateway port in firewalld: + +```shell +sudo firewall-cmd --add-port=8080/tcp --permanent +sudo firewall-cmd --reload +``` + +For localhost-only access (the default use case), no firewall changes +are needed. Loopback traffic is not filtered by firewalld. + +mTLS prevents unauthenticated access even when the port is open to the +network. + +## Common issues + +### "No active gateway" + +The CLI cannot find a registered gateway. This happens when the +gateway is running but has not been registered with the CLI. + +```shell +openshell gateway add --local https://127.0.0.1:8080 +``` + +### Gateway fails to start + +Check the journal for error details: + +```shell +journalctl --user -u openshell-gateway --no-pager -n 50 +``` + +Common causes: + +**cgroups v1 detected.** The Podman driver requires cgroups v2. +Check the version: + +```shell +stat -fc %T /sys/fs/cgroup +``` + +Expected output: `cgroup2fs`. If it shows `tmpfs`, enable cgroups v2: + +```shell +sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1" +sudo reboot +``` + +**Podman socket not available.** Ensure socket activation is enabled: + +```shell +systemctl --user enable --now podman.socket +systemctl --user status podman.socket +``` + +**TLS certificate errors.** If certs are corrupted, regenerate them: + +```shell +rm -rf ~/.local/state/openshell/tls +systemctl --user restart openshell-gateway +``` + +### Sandbox creation fails + +**subuid/subgid missing.** Rootless Podman requires subordinate +UID/GID ranges. If the journal shows warnings about `/etc/subuid` or +container creation fails: + +```shell +grep $USER /etc/subuid /etc/subgid +# If empty: +sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER +``` + +**Image pull failure.** Verify ghcr.io is reachable: + +```shell +podman pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest +``` + +### Images not updating + +The default image pull policy is `missing` -- images are pulled once +and cached. To update: + +```shell +podman pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest +podman pull ghcr.io/nvidia/openshell/supervisor:latest +``` + +Or set `OPENSHELL_SANDBOX_IMAGE_PULL_POLICY=always` in +`~/.config/openshell/gateway.env` and restart the gateway. + +### Gateway stops on logout + +Enable lingering so the service survives logout: + +```shell +sudo loginctl enable-linger $USER +``` + +## SELinux + +No SELinux configuration is required on stock Fedora or RHEL. The +Podman driver automatically applies the `:z` relabel option to TLS +bind mounts when SELinux is detected, allowing sandbox containers to +read the certificates through the MAC policy. + +## Upgrading + +After upgrading the RPM packages: + +```shell +sudo dnf update openshell openshell-gateway +systemctl --user restart openshell-gateway +``` + +The SQLite database schema is auto-migrated on startup. Running +sandboxes are stopped during the restart. + +The `gateway.env` file is not overwritten during upgrades. The +`init-gateway-env.sh` script is idempotent and only generates the file +on first start. New configuration options from newer versions can be +added manually by referencing CONFIGURATION.md or running +`openshell-gateway --help`. + +To pick up new container images after an upgrade: + +```shell +podman pull ghcr.io/nvidia/openshell/supervisor:latest +podman pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest +``` diff --git a/deploy/rpm/init-gateway-env.sh b/deploy/rpm/init-gateway-env.sh new file mode 100644 index 0000000000..299a190419 --- /dev/null +++ b/deploy/rpm/init-gateway-env.sh @@ -0,0 +1,115 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Generate the gateway environment configuration file on first start. +# +# Called from the systemd ExecStartPre directive to bootstrap the +# gateway configuration. Idempotent: exits immediately if the file +# already exists. +# +# Usage: +# init-gateway-env.sh +# +# The generated file contains an auto-generated SSH handshake secret +# and commented defaults for all gateway environment variables. + +set -euo pipefail + +ENV_FILE="${1:?Usage: init-gateway-env.sh }" + +# ── Idempotent: skip if env file already exists ───────────────────── +if [ -f "${ENV_FILE}" ]; then + exit 0 +fi + +# ── Create parent directory ───────────────────────────────────────── +mkdir -p "$(dirname "${ENV_FILE}")" + +# ── Generate SSH handshake secret ─────────────────────────────────── +SECRET=$(od -An -tx1 -N32 /dev/urandom | tr -dc 0-9a-f) + +# ── Write environment file ────────────────────────────────────────── +cat > "${ENV_FILE}" << EOF +# OpenShell Gateway Environment Configuration +# Generated on first start. Edit freely; this file is not overwritten. +# +# Run 'openshell-gateway --help' for the full list of options. +# See /usr/share/doc/openshell-gateway/ for guides. + +# ---- Required ---- + +# Shared secret for gateway-to-sandbox SSH handshake authentication. +# Auto-generated on first start. To regenerate: +# openssl rand -hex 32 +OPENSHELL_SSH_HANDSHAKE_SECRET=${SECRET} + +# ---- Optional (uncomment to override defaults) ---- + +# Database URL for gateway state persistence. +# Default for the user unit: sqlite://\$XDG_STATE_HOME/openshell/gateway.db +#OPENSHELL_DB_URL=sqlite:///path/to/gateway.db + +# Compute driver: podman (default for RPM), docker, kubernetes. +#OPENSHELL_DRIVERS=podman + +# Bind address. 0.0.0.0 listens on all interfaces; mTLS prevents +# unauthenticated access. +#OPENSHELL_BIND_ADDRESS=0.0.0.0 + +# API port (default: 8080). +#OPENSHELL_SERVER_PORT=8080 + +# Log level: trace, debug, info, warn, error. +#OPENSHELL_LOG_LEVEL=info + +# ---- Images ---- + +# Supervisor binary OCI image (mounted read-only into sandboxes). +#OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/nvidia/openshell/supervisor:latest + +# Default sandbox base image. +#OPENSHELL_SANDBOX_IMAGE=ghcr.io/nvidia/openshell-community/sandboxes/base:latest + +# Image pull policy: always, missing (default), never, newer. +# Use 'always' to pick up new tags automatically. +# Use 'never' for air-gapped environments with pre-loaded images. +#OPENSHELL_SANDBOX_IMAGE_PULL_POLICY=missing + +# ---- TLS (mTLS enabled by default) ---- +# PKI is auto-generated by init-pki.sh on first start. Client certs are +# placed in ~/.config/openshell/gateways/openshell/mtls/ so the CLI +# discovers them automatically. +# +# To use externally-managed certs, uncomment and edit the paths below. +# To rotate certs, delete ~/.local/state/openshell/tls/ and restart. +# WARNING: Disabling TLS with the default bind address (0.0.0.0) exposes +# the gateway API to the entire network with NO authentication. Only +# disable TLS when behind a TLS-terminating reverse proxy, or restrict +# OPENSHELL_BIND_ADDRESS to 127.0.0.1. +#OPENSHELL_DISABLE_TLS=true + +# Server TLS (gateway listens with these certs). +#OPENSHELL_TLS_CERT=\$XDG_STATE_HOME/openshell/tls/server/tls.crt +#OPENSHELL_TLS_KEY=\$XDG_STATE_HOME/openshell/tls/server/tls.key +#OPENSHELL_TLS_CLIENT_CA=\$XDG_STATE_HOME/openshell/tls/ca.crt + +# Podman driver: client certs bind-mounted into sandbox containers. +#OPENSHELL_PODMAN_TLS_CA=\$XDG_STATE_HOME/openshell/tls/ca.crt +#OPENSHELL_PODMAN_TLS_CERT=\$XDG_STATE_HOME/openshell/tls/client/tls.crt +#OPENSHELL_PODMAN_TLS_KEY=\$XDG_STATE_HOME/openshell/tls/client/tls.key + +# ---- Podman driver ---- + +# Podman API Unix socket path. +#OPENSHELL_PODMAN_SOCKET=\$XDG_RUNTIME_DIR/podman/podman.sock + +# Podman bridge network name for sandbox containers. +#OPENSHELL_NETWORK_NAME=openshell + +# Container stop timeout in seconds (SIGTERM then SIGKILL). +#OPENSHELL_STOP_TIMEOUT=10 +EOF + +chmod 600 "${ENV_FILE}" +echo "Gateway environment generated: ${ENV_FILE}" diff --git a/deploy/rpm/init-pki.sh b/deploy/rpm/init-pki.sh new file mode 100755 index 0000000000..900ec20a6a --- /dev/null +++ b/deploy/rpm/init-pki.sh @@ -0,0 +1,197 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Generate a self-signed PKI bundle for the OpenShell gateway. +# +# Called from the systemd ExecStartPre directive to bootstrap mTLS on +# first start. Idempotent: exits immediately if all cert files exist. +# Detects and recovers from partial PKI state (e.g. interrupted runs). +# +# All files are generated in a temporary staging directory first and +# moved into place only after the full PKI is complete, preventing +# partial state from persisting across failures. +# +# Usage: +# init-pki.sh +# +# Output layout: +# /ca.crt CA certificate +# /ca.key CA private key (mode 0600) +# /server/tls.crt Server certificate +# /server/tls.key Server private key (mode 0600) +# /client/tls.crt Client certificate +# /client/tls.key Client private key (mode 0600) +# +# Client certs are also copied to the CLI's auto-discovery directory: +# $XDG_CONFIG_HOME/openshell/gateways/openshell/mtls/{ca.crt,tls.crt,tls.key} + +set -euo pipefail + +PKI_DIR="${1:?Usage: init-pki.sh }" + +# ── Resolve CLI cert directory ─────────────────────────────────────── +CLI_MTLS_DIR="${XDG_CONFIG_HOME:-${HOME}/.config}/openshell/gateways/openshell/mtls" + +# ── Required PKI files ─────────────────────────────────────────────── +PKI_FILES=( + "${PKI_DIR}/ca.crt" + "${PKI_DIR}/ca.key" + "${PKI_DIR}/server/tls.crt" + "${PKI_DIR}/server/tls.key" + "${PKI_DIR}/client/tls.crt" + "${PKI_DIR}/client/tls.key" +) + +CLI_FILES=( + "${CLI_MTLS_DIR}/ca.crt" + "${CLI_MTLS_DIR}/tls.crt" + "${CLI_MTLS_DIR}/tls.key" +) + +# ── Idempotent: skip if all PKI files exist ────────────────────────── +all_pki_exist=true +for f in "${PKI_FILES[@]}"; do + if [ ! -f "$f" ]; then + all_pki_exist=false + break + fi +done + +if [ "$all_pki_exist" = true ]; then + # PKI is complete. Ensure CLI copies also exist (they may have been + # deleted independently, e.g. user cleared their config directory). + cli_ok=true + for f in "${CLI_FILES[@]}"; do + if [ ! -f "$f" ]; then + cli_ok=false + break + fi + done + if [ "$cli_ok" = false ]; then + echo "PKI exists but CLI auto-discovery certs missing; re-copying..." + mkdir -p "${CLI_MTLS_DIR}" + cp "${PKI_DIR}/ca.crt" "${CLI_MTLS_DIR}/ca.crt" + cp "${PKI_DIR}/client/tls.crt" "${CLI_MTLS_DIR}/tls.crt" + cp "${PKI_DIR}/client/tls.key" "${CLI_MTLS_DIR}/tls.key" + chmod 600 "${CLI_MTLS_DIR}/tls.key" + fi + exit 0 +fi + +# ── Partial state recovery ─────────────────────────────────────────── +# If some PKI files exist but not all, a previous run was interrupted. +# Remove the partial state so we can regenerate cleanly. +partial=false +for f in "${PKI_FILES[@]}"; do + if [ -f "$f" ]; then + partial=true + break + fi +done +if [ "$partial" = true ]; then + echo "WARNING: Partial PKI detected in ${PKI_DIR}, regenerating..." + rm -f "${PKI_DIR}/ca.crt" "${PKI_DIR}/ca.key" "${PKI_DIR}/ca.srl" + rm -rf "${PKI_DIR}/server" "${PKI_DIR}/client" +fi + +# ── Temporary workspace (cleaned up on exit) ───────────────────────── +WORK=$(mktemp -d) +trap 'rm -rf "${WORK}"' EXIT + +# Stage directory mirrors the final PKI layout. +STAGE="${WORK}/pki" +mkdir -p "${STAGE}/server" "${STAGE}/client" + +# ── Server certificate SANs ───────────────────────────────────────── +# These must match what the supervisor connects to. The CLI also +# connects using localhost/127.0.0.1 by default. +cat > "${WORK}/server-san.cnf" <<'EOF' +[req] +distinguished_name = req_dn +req_extensions = v3_req +prompt = no + +[req_dn] +O = openshell +CN = openshell-server + +[v3_req] +subjectAltName = @alt_names + +[alt_names] +DNS.1 = localhost +DNS.2 = openshell +DNS.3 = openshell.openshell.svc +DNS.4 = openshell.openshell.svc.cluster.local +DNS.5 = host.containers.internal +DNS.6 = host.docker.internal +IP.1 = 127.0.0.1 +EOF + +# ── Generate CA (into staging) ─────────────────────────────────────── +openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \ + -keyout "${STAGE}/ca.key" \ + -out "${STAGE}/ca.crt" \ + -days 3650 -nodes \ + -subj "/O=openshell/CN=openshell-ca" \ + 2>/dev/null +chmod 600 "${STAGE}/ca.key" + +# ── Generate server certificate (into staging) ─────────────────────── +openssl req -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \ + -keyout "${STAGE}/server/tls.key" \ + -out "${WORK}/server.csr" \ + -nodes \ + -config "${WORK}/server-san.cnf" \ + 2>/dev/null + +openssl x509 -req \ + -in "${WORK}/server.csr" \ + -CA "${STAGE}/ca.crt" -CAkey "${STAGE}/ca.key" -CAcreateserial \ + -out "${STAGE}/server/tls.crt" \ + -days 3650 \ + -extensions v3_req \ + -extfile "${WORK}/server-san.cnf" \ + 2>/dev/null +chmod 600 "${STAGE}/server/tls.key" + +# ── Generate client certificate (into staging) ─────────────────────── +openssl req -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \ + -keyout "${STAGE}/client/tls.key" \ + -out "${WORK}/client.csr" \ + -nodes \ + -subj "/O=openshell/CN=openshell-client" \ + 2>/dev/null + +openssl x509 -req \ + -in "${WORK}/client.csr" \ + -CA "${STAGE}/ca.crt" -CAkey "${STAGE}/ca.key" -CAcreateserial \ + -out "${STAGE}/client/tls.crt" \ + -days 3650 \ + 2>/dev/null +chmod 600 "${STAGE}/client/tls.key" + +# ── Move staged PKI into final location ────────────────────────────── +# Create parent directories and move files individually. Using mv on +# individual files rather than whole directories so we do not clobber +# the target directory if it already exists. +mkdir -p "${PKI_DIR}/server" "${PKI_DIR}/client" +mv "${STAGE}/ca.crt" "${PKI_DIR}/ca.crt" +mv "${STAGE}/ca.key" "${PKI_DIR}/ca.key" +mv "${STAGE}/server/tls.crt" "${PKI_DIR}/server/tls.crt" +mv "${STAGE}/server/tls.key" "${PKI_DIR}/server/tls.key" +mv "${STAGE}/client/tls.crt" "${PKI_DIR}/client/tls.crt" +mv "${STAGE}/client/tls.key" "${PKI_DIR}/client/tls.key" + +# ── Copy client certs to CLI auto-discovery directory ──────────────── +# The CLI automatically looks for certs at: +# $XDG_CONFIG_HOME/openshell/gateways//mtls/{ca.crt,tls.crt,tls.key} +# For localhost gateways, defaults to "openshell". +mkdir -p "${CLI_MTLS_DIR}" +cp "${PKI_DIR}/ca.crt" "${CLI_MTLS_DIR}/ca.crt" +cp "${PKI_DIR}/client/tls.crt" "${CLI_MTLS_DIR}/tls.crt" +cp "${PKI_DIR}/client/tls.key" "${CLI_MTLS_DIR}/tls.key" +chmod 600 "${CLI_MTLS_DIR}/tls.key" + +echo "PKI bootstrap complete: ${PKI_DIR}" diff --git a/docs/.markdownlint-cli2.jsonc b/docs/.markdownlint-cli2.jsonc new file mode 100644 index 0000000000..04cfcf7e20 --- /dev/null +++ b/docs/.markdownlint-cli2.jsonc @@ -0,0 +1,10 @@ +{ + "config": { + // MDX pages get their title from Fern frontmatter, not a top-level H1. + "MD041": false, + // MDX uses JSX components (, , ...) that look like HTML. + "MD033": false, + // Published docs should label fenced code blocks for rendering and copy UX. + "MD040": true + } +} diff --git a/docs/CONTRIBUTING.mdx b/docs/CONTRIBUTING.mdx index 1fcaf322ac..fd539575f9 100644 --- a/docs/CONTRIBUTING.mdx +++ b/docs/CONTRIBUTING.mdx @@ -56,13 +56,15 @@ PRs that touch `docs/**` or `fern/**` are validated by `.github/workflows/branch - Published docs use Fern MDX under `docs/`. - Every page starts with YAML frontmatter. Use `title` and `description` on every page, then add page-level metadata like `sidebar-title`, `keywords`, and `position` when the page needs them. - Include the SPDX license header as YAML comments inside frontmatter: - ``` + + ```text --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 title: "Page Title" --- ``` + - Do not repeat the page title as a body H1. Fern renders the title from frontmatter. ### Frontmatter Template @@ -122,9 +124,11 @@ These patterns are common in LLM-generated text and erode trust with technical r - End every sentence with a period. - Use `code` formatting for CLI commands, file paths, flags, parameter names, and values. - Use `shell` code blocks for copyable CLI examples. Do not prefix commands with `$`: + ```shell - openshell gateway start + openshell gateway add http://127.0.0.1:18080 --local --name local ``` + - Use `text` code blocks for transcripts, log output, and examples that should not be copied verbatim. - Use tables for structured comparisons. Keep tables simple (no nested formatting). - Use Fern components like ``, ``, and `` for callouts, not bold text. @@ -157,14 +161,14 @@ Use these consistently: 5. Run `mise run pre-commit` to catch formatting issues. 6. Open a PR with `docs:` as the conventional commit type. -``` +```text docs: update gateway deployment instructions ``` If your doc change accompanies a code change, include both in the same PR and use the code change's commit type: -``` -feat(cli): add --gpu flag to gateway start +```text +feat(cli): add gateway registration flag ``` ## Reviewing Doc PRs diff --git a/docs/about/architecture.mdx b/docs/about/architecture.mdx index 0c701e6a96..e1d2ff7460 100644 --- a/docs/about/architecture.mdx +++ b/docs/about/architecture.mdx @@ -3,25 +3,24 @@ # SPDX-License-Identifier: Apache-2.0 title: "How OpenShell Works" sidebar-title: "How It Works" -description: "OpenShell architecture overview covering the gateway, sandbox, policy engine, and privacy router." +description: "OpenShell architecture overview covering the gateway, sandbox, and policy engine." keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Architecture" position: 2 --- -OpenShell runs inside a Docker container. Each sandbox is an isolated environment managed through the gateway. Four components work together to keep agents secure. +OpenShell runs sandboxes through a gateway that uses one of four compute platforms: Kubernetes, Docker, Podman, or the experimental MicroVM runtime. Each sandbox is an isolated environment managed through the gateway. Three components work together to keep agents secure. ![OpenShell architecture diagram](/assets/images/architecture.svg) ## Components -The following table describes each component and its role in the system: +The following table describes each component and its role in the system: -| Component | Role | -|---|---| -| **Gateway** | Control-plane API that coordinates sandbox lifecycle and state, acts as the auth boundary, and brokers requests across the platform. | -| **Sandbox** | Isolated runtime that includes container supervision and policy-enforced egress routing. | -| **Policy Engine** | Policy definition and enforcement layer for filesystem, network, and process constraints. Defense in depth enforces policies from the application layer down to infrastructure and kernel layers. | -| **Privacy Router** | Privacy-aware LLM routing layer that keeps sensitive context on sandbox compute and routes based on cost and privacy policy. | +| Component | Role | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Gateway** | Control-plane API that coordinates sandbox lifecycle and state, and acts as the auth boundary between clients and sandboxes. | +| **Sandbox** | Isolated runtime that includes container supervision and policy-enforced egress routing. | +| **Policy Engine** | Policy definition and enforcement layer for filesystem, network, process, and inference constraints. Defense in depth enforces policies from the application layer down to infrastructure and kernel layers. | ## How a Request Flows @@ -31,21 +30,34 @@ Every outbound connection from agent code passes through the same decision path: 2. The proxy inside the sandbox intercepts the connection and identifies which binary opened it. 3. If the target is `https://inference.local`, the proxy handles it as managed inference before policy evaluation. OpenShell strips sandbox-supplied credentials, injects the configured backend credentials, and forwards the request to the managed model endpoint. 4. For every other destination, the proxy queries the policy engine with the destination, port, and calling binary. -5. The policy engine returns one of two decisions: - - **Allow** - the destination and binary match a policy block. Traffic flows directly to the external service. - - **Deny** - no policy block matched. The connection is blocked and logged. +5. The policy engine returns one of the following decisions. Explicit deny takes precedence over allow, and allow takes precedence over implicit deny. -For REST endpoints with TLS termination enabled, the proxy also decrypts TLS and checks each HTTP request against per-method, per-path rules before allowing it through. + | Decision | When it applies | Outcome | + | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | + | **Explicit Deny** | A deny rule, or a hardening rule such as server-side request forgery (SSRF) protection, a blocked control-plane port, or a Layer 7 (L7) deny, rejects the request even when an allow rule would otherwise permit it. | The connection is blocked and logged. | + | **Allow** | The destination and binary match a policy block, and no deny rule applies. | Traffic flows directly to the external service. | + | **Implicit Deny** | No policy block matched. | The connection is blocked and logged. | + +For REST endpoints, the proxy auto-detects Transport Layer Security (TLS) by peeking the first bytes of each tunnel. When TLS is detected, the proxy terminates it transparently using a per-sandbox ephemeral certificate authority (CA) so each HTTP request can be checked against per-method, per-path rules before forwarding. To trust a corporate or internal CA for upstream traffic, add the PEM-encoded certificate to the sandbox container's trust store at image build time. + +## Gateway Lifecycle + +OpenShell preserves gateway state in the configured gateway database. Kubernetes deployments commonly use the Helm chart's persistent volume, while standalone gateway deployments can use local storage or Postgres. Sandbox state depends on the selected compute driver and the storage configured for sandbox workloads. + +## Observability + +The sandbox emits operational decisions as Open Cybersecurity Schema Framework (OCSF) v1.7.0 structured logs. The events cover network connections, HTTP requests, SSH sessions, process lifecycle, detection findings, and configuration changes. For event classes and the JSON Lines (JSONL) export format, refer to [OCSF JSON Export](/observability/ocsf-json-export). ## Deployment Modes -OpenShell can run locally, on a remote host, or behind a cloud proxy. The architecture is identical in all cases — only the Docker container location and authentication mode change. +OpenShell supports four deployment modes: Kubernetes, Docker, Podman, and MicroVM. Each mode runs or exposes a gateway, then registers that endpoint with the CLI. The gateway's compute driver determines where sandboxes run. -| Mode | Description | Command | -|---|---|---| -| **Local** | The gateway runs inside Docker on your workstation. The CLI provisions it automatically on first use. | `openshell gateway start` | -| **Remote** | The gateway runs on a remote host via SSH. Only Docker is required on the remote machine. | `openshell gateway start --remote user@host` | -| **Cloud** | A gateway already running behind a reverse proxy (e.g. Cloudflare Access). Register and authenticate via browser. | `openshell gateway add https://gateway.example.com` | +| Mode | Description | +| -------------- | -------------------------------------------------------------------------------- | +| **Docker** | The gateway uses Docker to create local sandbox containers. | +| **Podman** | The gateway uses a Podman-compatible container runtime, commonly rootless. | +| **Kubernetes** | The Helm chart deploys the gateway and configures the Kubernetes compute driver. | +| **MicroVM** | The gateway uses the experimental VM compute driver. | You can register multiple gateways and switch between them with `openshell gateway select`. For the full deployment and management workflow, refer to the [Gateways](/sandboxes/manage-gateways) section. diff --git a/docs/get-started/quickstart.mdx b/docs/get-started/quickstart.mdx index 2f26c7bfba..9031338767 100644 --- a/docs/get-started/quickstart.mdx +++ b/docs/get-started/quickstart.mdx @@ -2,18 +2,20 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 title: "Quickstart" -description: "Install the OpenShell CLI and create your first sandboxed AI agent in two commands." -keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Installation, Quickstart" +description: "Install the OpenShell CLI, connect to a gateway, and create your first sandboxed AI agent." +keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Installation, Quickstart, Gateway, Docker, Kubernetes, Podman" position: 1 --- -This page gets you from zero to a running, policy-enforced sandbox in two commands. +This page gets you from a reachable OpenShell gateway to a running, policy-enforced sandbox. ## Prerequisites Before you begin, make sure you have: -- Docker Desktop running on your machine. +- A reachable OpenShell gateway. +- At least one compute platform configured for the gateway: Kubernetes, Docker, Podman, or MicroVM. +- The OpenShell CLI installed on your workstation. For a complete list of requirements, refer to [Support Matrix](/reference/support-matrix). @@ -31,11 +33,10 @@ If you prefer [uv](https://docs.astral.sh/uv/): uv tool install -U openshell ``` -After installing the CLI, run `openshell --help` in your terminal to see the full CLI reference, including all commands and flags. +After installing the CLI, run `openshell --help` in your terminal to see the full CLI reference. You can also clone the [NVIDIA OpenShell GitHub repository](https://github.com/NVIDIA/OpenShell) and use the `/openshell-cli` skill to load the CLI reference into your agent. - ## Create Your First OpenShell Sandbox @@ -100,8 +101,8 @@ Each definition bundles a container image, a tailored policy, and optional skill -Use the `--from` flag to pull other OpenShell sandbox images from the [NVIDIA Container Registry](https://registry.nvidia.com/). -For example, to pull the `base` image, run the following command: +Use the `--from` flag to pull other OpenShell sandbox images from the [OpenShell Community](https://github.com/NVIDIA/OpenShell-Community) catalog. +For example, to pull the `base` image, run: ```shell openshell sandbox create --from base @@ -110,46 +111,3 @@ openshell sandbox create --from base - -## Deploy a Gateway (Optional) - -Running `openshell sandbox create` without a gateway auto-bootstraps a local one. -To start the gateway explicitly or deploy to a remote host, choose the tab that matches your setup. - - - - - -Deploy an OpenShell gateway on Brev by clicking **Deploy** on the [OpenShell Launchable](https://brev.nvidia.com/launchable/deploy/now?launchableID=env-3Ap3tL55zq4a8kew1AuW0FpSLsg). - - - -After the instance starts running, find the gateway URL in the Brev console under **Using Secure Links**. -Copy the shareable URL for **port 8080**, which is the gateway endpoint. - -```shell -openshell gateway add https://.brevlab.com -openshell status -``` - - - - - - -Set up your Spark with NVIDIA Sync first, or make sure SSH access is configured (such as SSH keys added to the host). - - - -Deploy to a DGX Spark machine over SSH: - -```shell -openshell gateway start --remote @.local -openshell status -``` - -After `openshell status` shows the gateway as healthy, all subsequent commands route through the SSH tunnel. - - - - diff --git a/docs/tutorials/first-network-policy.mdx b/docs/get-started/tutorials/first-network-policy.mdx similarity index 98% rename from docs/tutorials/first-network-policy.mdx rename to docs/get-started/tutorials/first-network-policy.mdx index bb68e6135b..effe7c445d 100644 --- a/docs/tutorials/first-network-policy.mdx +++ b/docs/get-started/tutorials/first-network-policy.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "Write Your First Sandbox Network Policy" sidebar-title: "First Network Policy" -slug: "tutorials/first-network-policy" +slug: "get-started/tutorials/first-network-policy" description: "See how OpenShell network policies work by creating a sandbox, observing default-deny in action, and applying a fine-grained L7 read-only rule." keywords: "Generative AI, Cybersecurity, Tutorial, Policy, Network Policy, Sandbox, Security" --- @@ -218,4 +218,4 @@ bash examples/sandbox-policy-quickstart/demo.sh ## Next Steps -- To walk through a full policy iteration with Claude Code, including diagnosing denials and applying fixes from outside the sandbox, refer to [Github Sandbox](/tutorials/github-sandbox). +- To walk through a full policy iteration with Claude Code, including diagnosing denials and applying fixes from outside the sandbox, refer to [GitHub Sandbox](/get-started/tutorials/github-sandbox). diff --git a/docs/tutorials/github-sandbox.mdx b/docs/get-started/tutorials/github-sandbox.mdx similarity index 95% rename from docs/tutorials/github-sandbox.mdx rename to docs/get-started/tutorials/github-sandbox.mdx index 0490b6bc1f..1f225fb600 100644 --- a/docs/tutorials/github-sandbox.mdx +++ b/docs/get-started/tutorials/github-sandbox.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "Grant GitHub Push Access to a Sandboxed Agent" sidebar-title: "GitHub Push Access" -slug: "tutorials/github-sandbox" +slug: "get-started/tutorials/github-sandbox" description: "Learn the iterative policy workflow by launching a sandbox, diagnosing a GitHub access denial, and applying a custom policy to fix it." keywords: "Generative AI, Cybersecurity, Tutorial, GitHub, Sandbox, Policy, Claude Code" --- @@ -152,7 +152,7 @@ The following steps outline the expected process done by the agent: 1. Inspects the deny reasons. 2. Writes an updated policy that adds `github_git` and `github_api` blocks that grant write access to your repository. -3. Saves the policy to `/tmp/sandbox-policy-update.yaml`. +3. Saves the policy to `/tmp/sandbox-policy-update.yaml`. ## Review the Generated Policy @@ -247,17 +247,29 @@ network_policies: endpoints: - host: api.github.com port: 443 + path: "/repos///**" protocol: rest enforcement: enforce rules: - # GraphQL API (used by gh CLI) - - allow: - method: POST - path: "/graphql" # Full read-write access to the repository - allow: method: "*" path: "/repos///**" + - host: api.github.com + port: 443 + path: "/graphql" + protocol: graphql + enforcement: enforce + rules: + # GitHub GraphQL API (used by gh CLI) + - allow: + operation_type: query + - allow: + operation_type: mutation + fields: [createIssue, updateIssue, addComment] + deny_rules: + - operation_type: mutation + fields: [deleteRepository, deleteRef, updateBranchProtectionRule] binaries: - { path: /usr/local/bin/claude } - { path: /usr/local/bin/opencode } @@ -334,7 +346,7 @@ The push completes successfully. The `openshell term` dashboard now shows `l7_de ## Clean Up -When you are finished, delete the sandbox to free cluster resources: +When you are finished, delete the sandbox to free gateway compute resources: ```shell openshell sandbox delete @@ -348,4 +360,4 @@ The following resources cover related topics in greater depth: - To add per-repository access levels (read-write vs read-only) or restrict to specific API methods, refer to the [Policy Schema Reference](/reference/policy-schema). - To learn the full policy iteration workflow (pull, edit, push, verify), refer to [Policies](/sandboxes/policies). -- To inject credentials automatically instead of pasting tokens, refer to [Manage Providers](/sandboxes/manage-providers +- To inject credentials automatically instead of pasting tokens, refer to [Manage Providers](/sandboxes/manage-providers) diff --git a/docs/tutorials/index.mdx b/docs/get-started/tutorials/index.mdx similarity index 73% rename from docs/tutorials/index.mdx rename to docs/get-started/tutorials/index.mdx index 970633a887..f5a79b543b 100644 --- a/docs/tutorials/index.mdx +++ b/docs/get-started/tutorials/index.mdx @@ -2,7 +2,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 title: "Tutorials" -slug: "tutorials" +slug: "get-started/tutorials" description: "Step-by-step walkthroughs for OpenShell, from first sandbox to production-ready policies." keywords: "Generative AI, Cybersecurity, Tutorial, Sandbox, Policy" position: 1 @@ -12,22 +12,22 @@ Hands-on walkthroughs that teach OpenShell concepts by building real configurati - + Create a sandbox, observe default-deny networking, apply a read-only L7 policy, and inspect audit logs. No AI agent required. - + Launch Claude Code in a sandbox, diagnose a policy denial, and iterate on a custom GitHub policy from outside the sandbox. - + Route inference through Ollama using cloud-hosted or local models, and verify it from a sandbox. - + Route inference to a local LM Studio server via the OpenAI or Anthropic compatible APIs. diff --git a/docs/tutorials/inference-ollama.mdx b/docs/get-started/tutorials/inference-ollama.mdx similarity index 98% rename from docs/tutorials/inference-ollama.mdx rename to docs/get-started/tutorials/inference-ollama.mdx index 5140038fc7..4fcc018a7b 100644 --- a/docs/tutorials/inference-ollama.mdx +++ b/docs/get-started/tutorials/inference-ollama.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "Run Local Inference with Ollama" sidebar-title: "Inference with Ollama" -slug: "tutorials/inference-ollama" +slug: "get-started/tutorials/inference-ollama" description: "Run local and cloud models inside an OpenShell sandbox using the Ollama community sandbox, or route sandbox requests to a host-level Ollama server." keywords: "Generative AI, Cybersecurity, Tutorial, Inference Routing, Ollama, Local Inference, Sandbox" --- @@ -45,7 +45,7 @@ Chat with a local model ollama run qwen3.5 ``` -Or a cloud model +Or a cloud model ```shell ollama run kimi-k2.5:cloud @@ -77,7 +77,7 @@ ollama launch claude --yes --model qwen3.5 | No local GPU | `qwen3.5:cloud` | Runs on Ollama's cloud infrastructure, no `ollama pull` required | -Cloud models use the `:cloud` tag suffix and do not require local hardware. +Cloud models use the `:cloud` tag suffix and do not require local hardware. ```shell openshell sandbox create --from ollama diff --git a/docs/tutorials/local-inference-lmstudio.mdx b/docs/get-started/tutorials/local-inference-lmstudio.mdx similarity index 99% rename from docs/tutorials/local-inference-lmstudio.mdx rename to docs/get-started/tutorials/local-inference-lmstudio.mdx index fec37fe2db..976770d1fe 100644 --- a/docs/tutorials/local-inference-lmstudio.mdx +++ b/docs/get-started/tutorials/local-inference-lmstudio.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "Route Local Inference Requests to LM Studio" sidebar-title: "Local Inference with LM Studio" -slug: "tutorials/local-inference-lmstudio" +slug: "get-started/tutorials/local-inference-lmstudio" description: "Configure inference.local to route sandbox requests to a local LM Studio server running on the gateway host." keywords: "Generative AI, Cybersecurity, Tutorial, Inference Routing, LM Studio, Local Inference, Sandbox" --- @@ -43,6 +43,7 @@ irm https://lmstudio.ai/install.ps1 | iex And start llmster: + ```shell lms daemon up ``` @@ -64,6 +65,7 @@ If you're using llmster in headless mode, run `lms server start --bind 0.0.0.0`. In the LM Studio app, head to the Model Search tab to download a small model like Qwen3.5 2B. In the terminal, use the following command to download and load the model: + ```shell lms get qwen/qwen3.5-2b lms load qwen/qwen3.5-2b diff --git a/docs/index.mdx b/docs/index.mdx index 8c09fffb3e..73b8d1d0b1 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -2,7 +2,6 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 title: "NVIDIA OpenShell Developer Guide" -slug: "get-started" description: "OpenShell is the safe, private runtime for autonomous AI agents. Run agents in sandboxed environments that protect your data, credentials, and infrastructure." keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Inference Routing" position: 1 @@ -10,7 +9,6 @@ position: 1 import { BadgeLinks } from "./_components/BadgeLinks"; - with { } breaks MDX/acorn */} +{/*Terminal demo styles live in fern/main.css — inline