Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,5 +286,68 @@ Standalone local deployments start the gateway with a selected runtime such as
Docker, Podman, or VM. The CLI can register multiple gateways and switch between
them without changing the sandbox architecture.

## Workspace Namespace Modes (Kubernetes)

The Kubernetes driver maps workspaces to namespaces through the `workspace_mode`
configuration field (`WorkspaceMode` in `crates/openshell-driver-kubernetes/src/config.rs`).
The mode controls namespace resolution, resource naming, sandbox CR watching, SA
token authentication, and RBAC requirements.

| Mode | Namespace resolution | Resource name | Namespace lifecycle |
|---|---|---|---|
| **Shared** (default) | Single static namespace from config | `{workspace}--{name}` | None |
| **Managed** | `openshell-{gateway_id}-{workspace}` | bare sandbox name | Driver creates and deletes |
| **Operator** | Workspace name maps 1:1 to a pre-provisioned namespace | bare sandbox name | External (platform team) |

**Shared** renders all sandboxes into one configured namespace. Resource names
embed the workspace prefix for collision avoidance. No namespace lifecycle
management. RBAC uses a namespace-scoped Role.

**Managed** auto-creates a K8s namespace per workspace on first sandbox create.
Each new namespace receives a ServiceAccount and copies OpenShift SCC UID-range
and supplemental-group annotations from the gateway namespace when present. The
driver deletes the namespace when the last sandbox in it is removed
(`delete_namespace_if_empty`). Requires a non-empty `gateway_id` (validated as a
DNS-1123 label at startup) so the namespace prefix fits within the K8s 63-character
limit. RBAC promotes sandbox CRD permissions to a ClusterRole and adds namespace
`create`/`delete` and ServiceAccount `create`/`get` permissions.

**Operator** uses pre-provisioned namespaces discovered through two optional
sources: a K8s label selector (`operator_namespace_label`) and a drop-in
allowlist file (`operator_namespace_file`). At least one must be configured.
The `OperatorNamespaceAllowlist` (`Arc<RwLock<BTreeSet<String>>>`) is populated
at runtime by background watchers and read by the namespace resolver. Sandbox
creation fails closed if the workspace is not in the current allowlist. Platform
teams manage namespace lifecycle externally. RBAC uses the same ClusterRole as
managed mode but without namespace `create`/`delete` or ServiceAccount
permissions.

### Watching and Querying

Managed and operator modes set `is_multi_namespace() == true`, which switches
sandbox CR watchers from namespace-scoped `Api::namespaced` to cluster-wide
`Api::all_with`. In managed mode the driver scopes cluster-wide queries with a
`LABEL_GATEWAY_ID` label selector to support multiple gateways on the same
cluster. K8s Events are not watched in cluster-wide mode — the cluster-wide
watcher emits only sandbox CR changes, not platform events.

### SA Token Authentication

The gateway's `K8sServiceAccountAuthenticator` adapts its `NamespaceValidator`
per mode (`crates/openshell-server/src/auth/k8s_sa.rs`):

- **Shared:** `Exact` — accepts only the single configured namespace.
- **Managed:** `Prefix` — accepts any namespace starting with `openshell-{gateway_id}-`.
- **Operator:** `Allowlist` — accepts namespaces present in the dynamic
`BTreeSet` populated by the label/file watchers. Starts empty (fail-closed)
until the first watcher update.

### Credential Driver Integration

The Kubernetes Secrets credential driver (`openshell-driver-kubernetes-secrets`)
stores secrets in workspace-specific namespaces when `workspace_mode` is managed
or operator. In shared mode, all secrets render into the single configured
namespace.

When runtime infrastructure changes, validate the relevant sandbox e2e path and
update the matching driver README if a maintainer-facing constraint changes.
3 changes: 3 additions & 0 deletions crates/openshell-core/src/driver_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ pub const LABEL_SANDBOX_NAMESPACE: &str = "openshell.ai/sandbox-namespace";
/// Container/pod label carrying the sandbox workspace.
pub const LABEL_SANDBOX_WORKSPACE: &str = "openshell.ai/sandbox-workspace";

/// Label carrying the gateway identity on managed namespaces.
pub const LABEL_GATEWAY_ID: &str = "openshell.ai/gateway-id";

/// Label selector that matches all OpenShell-managed resources which carry a
/// sandbox ID label. Used by list and watch operations to exclude foreign
/// resources from the same namespace.
Expand Down
74 changes: 72 additions & 2 deletions crates/openshell-driver-kubernetes-secrets/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,42 @@ impl CredentialDriverService {
}
}

#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Deserialize)]
#[serde(rename_all = "snake_case")]
enum WorkspaceMode {
#[default]
Shared,
Managed,
Operator,
}

#[derive(Debug, Clone, PartialEq, Eq)]
struct KubernetesSecretsDriverSettings {
namespace: String,
allow_reference_namespace: bool,
workspace_mode: WorkspaceMode,
gateway_id: String,
}

impl KubernetesSecretsDriverSettings {
fn target_namespace(&self, workspace: &str) -> String {
match self.workspace_mode {
WorkspaceMode::Shared => self.namespace.clone(),
WorkspaceMode::Managed => {
format!("openshell-{}-{}", self.gateway_id, workspace)
}
WorkspaceMode::Operator => workspace.to_string(),
}
}
}

#[derive(Debug, Clone, Default, serde::Deserialize)]
#[serde(default, deny_unknown_fields)]
struct KubernetesSecretsDriverConfig {
namespace: Option<String>,
allow_reference_namespace: bool,
workspace_mode: WorkspaceMode,
gateway_id: Option<String>,
}

#[derive(Debug, Clone, PartialEq, Eq)]
Expand Down Expand Up @@ -134,7 +159,10 @@ impl KubernetesSecretsCredentialDriver {
credential_key: &str,
) -> Result<KubernetesSecretReference, Status> {
let reference = Self::parse_handle(handle, credential_key)?;
if reference.namespace != self.settings.namespace
// In managed/operator modes secrets live in workspace-specific
// namespaces so cross-namespace handles are expected.
if self.settings.workspace_mode == WorkspaceMode::Shared
&& reference.namespace != self.settings.namespace
&& !self.settings.allow_reference_namespace
{
return Err(Status::permission_denied(format!(
Expand Down Expand Up @@ -175,7 +203,7 @@ impl KubernetesSecretsCredentialDriver {
reference
} else {
KubernetesSecretReference {
namespace: self.settings.namespace.clone(),
namespace: self.settings.target_namespace(&request.workspace),
secret_name: managed_secret_name(
&request.workspace,
&request.provider_id,
Expand Down Expand Up @@ -508,6 +536,8 @@ impl KubernetesSecretsDriverSettings {
Ok(Self {
namespace,
allow_reference_namespace: config.allow_reference_namespace,
workspace_mode: config.workspace_mode,
gateway_id: config.gateway_id.unwrap_or_default(),
})
}
}
Expand Down Expand Up @@ -842,6 +872,8 @@ mod tests {
let settings = KubernetesSecretsDriverSettings {
namespace: "openshell".to_string(),
allow_reference_namespace: false,
workspace_mode: WorkspaceMode::Shared,
gateway_id: String::new(),
};
let reference = KubernetesSecretsCredentialDriver::parse_handle(
&handle("v1:other-namespace:provider-secret"),
Expand All @@ -865,6 +897,8 @@ mod tests {
let settings = KubernetesSecretsDriverSettings {
namespace: "openshell".to_string(),
allow_reference_namespace: true,
workspace_mode: WorkspaceMode::Shared,
gateway_id: String::new(),
};
let reference = KubernetesSecretsCredentialDriver::parse_handle(
&handle("v1:other-namespace:provider-secret"),
Expand Down Expand Up @@ -1065,4 +1099,40 @@ mod tests {
assert_eq!(err.code(), Code::FailedPrecondition);
assert!(err.message().contains("is not managed by OpenShell"));
}

#[test]
fn target_namespace_shared_returns_static_namespace() {
let settings = KubernetesSecretsDriverSettings {
namespace: "openshell".to_string(),
allow_reference_namespace: false,
workspace_mode: WorkspaceMode::Shared,
gateway_id: String::new(),
};
assert_eq!(settings.target_namespace("team-a"), "openshell");
assert_eq!(settings.target_namespace("team-b"), "openshell");
}

#[test]
fn target_namespace_managed_computes_from_workspace() {
let settings = KubernetesSecretsDriverSettings {
namespace: "openshell".to_string(),
allow_reference_namespace: false,
workspace_mode: WorkspaceMode::Managed,
gateway_id: "gw1".to_string(),
};
assert_eq!(settings.target_namespace("team-a"), "openshell-gw1-team-a");
assert_eq!(settings.target_namespace("team-b"), "openshell-gw1-team-b");
}

#[test]
fn target_namespace_operator_uses_workspace_name() {
let settings = KubernetesSecretsDriverSettings {
namespace: "openshell".to_string(),
allow_reference_namespace: false,
workspace_mode: WorkspaceMode::Operator,
gateway_id: String::new(),
};
assert_eq!(settings.target_namespace("team-a"), "team-a");
assert_eq!(settings.target_namespace("prod-ns"), "prod-ns");
}
}
14 changes: 12 additions & 2 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,18 @@
Kubernetes-backed compute driver for OpenShell cluster deployments.

The driver uses the Kubernetes API to create, delete, fetch, and watch sandbox
custom resources in the configured namespace. It runs in-process with the
gateway server.
custom resources. It runs in-process with the gateway server and supports three
workspace namespace modes via `workspace_mode`:

- **Shared** (default): All sandboxes render into a single static namespace.
Resource names use `{workspace}--{name}` for collision avoidance.
- **Managed**: The driver auto-creates/deletes a K8s namespace per workspace
(`openshell-{gateway_id}-{workspace_name}`), creates a ServiceAccount in each,
and copies OpenShift SCC annotations from the gateway namespace when present.
- **Operator**: Workspace names map 1:1 to pre-provisioned namespaces discovered
via label selector (`operator_namespace_label`) and/or drop-in allowlist file
(`operator_namespace_file`). Sandbox creation fails closed if the workspace
namespace is not in the current allowlist.

## Runtime Model

Expand Down
Loading
Loading