Configurable Container Runtime (Docker/Podman)

[Hadar301]

Jira: https://redhat.atlassian.net/browse/APPENG-6041

---

Summary

Makes the container runtime inside the gateway VM configurable via a single Helm value (containerRuntime: docker|podman). Ships two pre-built golden images — a Docker variant for NemoClaw and a Podman variant for openclaw/opencode — and routes all runtime-specific behavior through the toggle. Switching runtimes requires only a values change + VM recreate; no golden image rebuild is needed.

---

What changed

charts/openshell-saw

• values.yaml — new containerRuntime: docker field (default); source.dataSource cleared so it auto-derives from the runtime
• templates/_helpers.tpl — new openshell-sandbox.dataSourceName helper: resolves openshell-gateway-docker for docker, openshell-gateway for podman, with explicit override support
• templates/virtualmachine.yaml — DataSource name now uses the helper
• templates/cloudinit-sandbox.yaml — OPENSHELL_DRIVERS set from containerRuntime instead of hardcoded docker
• templates/configmap-scripts.yaml — multiple changes across all setup scripts:
  • RUNTIME variable injected at the top of run-setup.sh, setup-nemoclaw.sh, setup-dashboard.sh, setup-workspaces.sh
  • All docker pull/create/cp/rm/run/stop/logs replaced with ${RUNTIME}
  • Registry auth block (docker login, insecure-registries daemon config) wrapped in a Docker-only conditional; Podman path prints a clear skip message
  • BUILD_NS and NEMOCLAW_CLI_IMAGE gated so they don't cause unbound-variable errors on the Podman path
  • NemoClaw+Podman validation guard: setup exits immediately with a clear error if containerRuntime=podman and onboardCli=nemoclaw
  • setup-dashboard.sh exits gracefully (not with exit 1) when oidc.issuerUrl is not configured, preventing spurious Job retries in standalone deployments without Keycloak
  • models.mode in the openclaw config set to replace (only configured providers shown in TUI) instead of the invalid custom

image-builder-charts/helm/openshell-gateway-image

• values.yaml — new containerRuntime: docker field
• templates/buildconfig.yaml — branches on containerRuntime:
  • Docker: removes Podman, installs Docker CE, enables docker.service, derives GRPC endpoint from docker network inspect bridge (172.17.0.1)
  • Podman: keeps Fedora-default Podman, enables user-level podman.socket (after the user systemd bus is ready), sets OPENSHELL_PODMAN_SOCKET in gateway.env, derives GRPC endpoint from podman network inspect podman (10.88.0.1)
  • $imageName derived from runtime: openshell-gateway-docker (Docker) or openshell-gateway (Podman)
• templates/golden-image.yaml and templates/imagestream.yaml — use $imageName for all resource names

Makefile and scripts

• Makefile-quickstart — CONTAINER_RUNTIME and IMAGE_NAME variables; build-openshell-gateway uses IMAGE_NAME throughout; openshell-saw-create forwards CONTAINER_RUNTIME and GOVERNANCE_ENABLED; configure-gateway is now idempotent (removes stale entry before re-adding)
• scripts/openshell-saw-create.sh — accepts CONTAINER_RUNTIME and GOVERNANCE_ENABLED env vars, passes both to helm
• scripts/e2e-test.sh (line 304) and tests/test-bootc-e2e.sh (line 346) — runtime version check uses the configured runtime instead of the stale podman --version

macOS copy-images fix

make copy-images fails on macOS because podman runs inside a VM and large-layer TCP uploads to the OpenShift external registry route are dropped mid-transfer. On uname -s == Darwin, the target now delegates to an in-cluster skopeo Job that pushes directly to image-registry.openshift-image-registry.svc:5000, bypassing the external route entirely.

New files: scripts/mirror-images-incluster.sh, scripts/mirror-images-rbac.yaml, scripts/mirror-images-job.yaml

Docs

• docs/container-runtime.md — new doc covering: runtime comparison table, golden image build commands, deploy commands, TUI connection steps (mTLS cert copy, port-forward, ssh-proxy), and known limitations (NemoClaw incompatibility, inference.local, rootless Podman caveats)

---

E2E validation (Docker path, clean first-time install)

Tested against cluster-7brvw.dyn.redhatworkshops.io:

Step	Result
make build-openshell-gateway CONTAINER_RUNTIME=docker	✅ openshell-gateway-docker ImageStream + DataSource created
CDI golden image import	✅ DataVolume Succeeded
Setup Job — DataSource: openshell-gateway-docker	✅
Setup Job — docker login to internal registry	✅
Setup Job — docker pull/create/cp/rm binary extraction	✅
Setup Job — nemoclaw CLI installed	✅
Setup Job — sandbox created from nemoclaw-sandbox:latest	✅
Setup Job — openclaw gateway ready	✅
Setup Job — dashboard skipped cleanly (no Keycloak)	✅
run-create OK / Setup complete	✅
TUI — openclaw responded to prompts	✅

Podman path validation:

• NemoClaw preflight: ✗ Docker is not reachable — correct expected failure
• Binary extraction via podman pull/create/cp/rm — confirmed in logs
• OPENSHELL_DRIVERS=podman in cloud-init — confirmed
• DataSource: openshell-gateway — confirmed

---

Notes for reviewers

• NemoClaw + openshell 0.0.99: NemoClaw's inference.local routing (nemoclaw onboard step 4) fails with externally-supervised gateways running 0.0.99+. The setup Job's fallback creates the inference provider and sandbox via openshell provider create + openshell sandbox create, which is sufficient for TUI use. This is a known upstream limitation unrelated to this PR.
• nemoclaw-sandbox image: Must be available in the internal registry before the setup Job runs. On first deploy, use make copy-images (Linux) or it runs automatically on macOS via the new in-cluster Job path.
• TUI mTLS: The gateway cert is valid only for 127.0.0.1. TUI access requires copying the client cert from the VM (virtctl scp) and port-forwarding the gateway service. Steps documented in docs/container-runtime.md. A follow-up improvement would be publishing the client cert as a k8s Secret from the setup Job to eliminate the virtctl dependency.

[sauagarwa]

Good PR overall — clean single-toggle design, solid guard rails. A few things to address:

1. In-cluster mirror Job: use oc image mirror instead of skopeo

The Job already runs inside the cluster, so oc image mirror can push directly to image-registry.openshift-image-registry.svc:5000 — same as skopeo but without introducing a new tool. This lets you:

• Use registry.redhat.io/openshift4/ose-cli instead of registry.redhat.io/ubi8/skopeo
• Drop the anyuid SCC grant (oc image mirror doesn't need root)
• Keep tooling consistent with the Linux path

Applies to: scripts/mirror-images-job.yaml, scripts/mirror-images-incluster.sh, scripts/mirror-images-rbac.yaml

2. Podman socket wait loop needs a warning on timeout

In buildconfig.yaml, the Podman socket wait loop silently continues if the socket never appears after 15s. If the gateway then can't reach Podman, the failure surfaces later with no clue about the root cause. Add a warning:

for _ in $(seq 1 15); do
  if [[ -S "/run/user/${USER_UID}/podman/podman.sock" ]]; then break; fi
  sleep 1
done
if [[ ! -S "/run/user/${USER_UID}/podman/podman.sock" ]]; then
  echo "WARNING: Podman socket not found after 15s — gateway may fail to start" >&2
fi

3. models.mode = 'replace' is unrelated to the runtime toggle

The fix in configmap-scripts.yaml (changing the invalid custom to replace) is correct but unrelated to container runtime configurability. Consider splitting it into a separate commit for cleaner bisect history.