bug: containerized Podman gateway broken on Fedora 44 (SELinux, UID mapping, port binding, state path)

[TomasTomecek]

Agent Diagnostic

Loaded the debug-openshell-cluster skill and investigated a live Fedora 44 host running the gateway as a rootless Podman container. Both the gateway and sandbox containers had exited ~21 hours prior.

Skills loaded: debug-openshell-cluster

Investigation steps and findings:

1. openshell status → Connection refused on http://127.0.0.1:8080 — gateway container had exited (code 0, clean shutdown signal)
2. podman logs openshell-gateway → Sandbox containers were immediately exiting with code 1; gateway log showed sandbox JWT auth not configured and a subuid/subgid warning
3. Inspected sandbox env → OPENSHELL_ENDPOINT=http://host.containers.internal:8080
4. Inspected gateway container → port mapped to 127.0.0.1:8080 only; sandbox on openshell bridge (10.89.1.0/24) resolves host.containers.internal to 10.89.1.1 — port 8080 not listening there → Connection refused
5. Recreated gateway with -p 0.0.0.0:8080:8080 → exited with Permission denied on Podman socket
6. Gateway image now runs as UID 1000:1000; without --userns=keep-id, container UID 1000 maps to host UID 100999, not socket owner tt (1000)
7. Added --userns=keep-id → DB volume files were owned by host UID 100999 (written by old container without keep-id) → unable to open database file
8. Fixed with podman unshare chown -R 0:0 on volume data → gateway started but hit SELinux AVC denial: container_t denied write on user_tmp_t sock_file
9. Added --security-opt label=disable → gateway reached Podman driver init; subuid warning fired (false positive — the check runs inside the container which doesn't have host's /etc/subuid mounted, but the host does have valid entries)
10. Gateway started but sandbox creation failed: "create sandbox token directory failed" — gateway tries to write to /app/.local/state/openshell/podman-sandbox-tokens/ inside container; that path is not writable by UID 1000 in the image
11. Set XDG_STATE_HOME=/run/user/1000/openshell-state with a mirrored bind mount (-v /run/user/1000/openshell-state:/run/user/1000/openshell-state) → token file written but Podman API returned 500: statfs ... sandbox.jwt: no such file or directory; gateway passes the container-internal path to the Podman API, which looks for it on the host — paths diverge unless source == destination in the bind mount
12. Used mirrored mount (host path == container path) → gateway wrote token, Podman found it, sandbox came up Ready
13. Also required: generating an Ed25519 JWT signing key pair and TOML config ([openshell.gateway.gateway_jwt] + [openshell.gateway.auth] allow_unauthenticated_users = true) — neither is documented for the Podman-in-container case

Description

I was following the official documentation at https://docs.nvidia.com/openshell/home. Happy to contribute improvements so that the next people using Fedora or SELinux won't run into these.

Actual behavior: Containerized Podman gateway fails to start or create sandboxes on Fedora 44 due to a cascade of issues: wrong port binding, UID/SELinux mismatches between container and host, missing sandbox JWT configuration, and a state-directory path mismatch between what the gateway writes and what the Podman API can find on the host.

Expected behavior: The documented podman run invocation for the gateway works out of the box on Fedora 44 with rootless Podman and SELinux enforcing, or the documentation covers each required flag and config step.

Reproduction Steps

1. Fedora 44 host, rootless Podman, SELinux enforcing
2. Run the gateway container as documented (or as shown in issue logs) with -p 127.0.0.1:8080:8080 and no --userns or SELinux flags
3. Attempt openshell sandbox create

Full working invocation (reached after debugging):

# Generate JWT signing keys (one-time)
mkdir -p ~/.config/openshell/jwt
openssl genpkey -algorithm ed25519 -out ~/.config/openshell/jwt/signing.pem
openssl pkey -in ~/.config/openshell/jwt/signing.pem -pubout -out ~/.config/openshell/jwt/public.pem
uuidgen > ~/.config/openshell/jwt/kid

# TOML config
cat > ~/.config/openshell/gateway.toml <<'TOML'
[openshell.gateway.auth]
allow_unauthenticated_users = true

[openshell.gateway.gateway_jwt]
signing_key_path = "/etc/openshell/jwt/signing.pem"
public_key_path  = "/etc/openshell/jwt/public.pem"
kid_path         = "/etc/openshell/jwt/kid"
gateway_id       = "openshell"
TOML

# State dir must be mirrored (same path on host and in container)
mkdir -p /run/user/1000/openshell-state

podman run -d --name openshell-gateway \
  --userns=keep-id \
  --security-opt label=disable \
  -p 0.0.0.0:8080:8080 \
  -v openshell-state:/var/openshell \
  -v /run/user/1000/podman/podman.sock:/var/run/podman.sock \
  -v ~/.config/openshell/jwt:/etc/openshell/jwt:ro \
  -v ~/.config/openshell/gateway.toml:/etc/openshell/gateway.toml:ro \
  -v /run/user/1000/openshell-state:/run/user/1000/openshell-state \
  -e OPENSHELL_DRIVERS=podman \
  -e OPENSHELL_PODMAN_SOCKET=/var/run/podman.sock \
  -e OPENSHELL_DB_URL=sqlite:/var/openshell/openshell.db \
  -e OPENSHELL_DISABLE_TLS=true \
  -e OPENSHELL_GATEWAY_CONFIG=/etc/openshell/gateway.toml \
  -e XDG_STATE_HOME=/run/user/1000/openshell-state \
  ghcr.io/nvidia/openshell/gateway:latest \
  --log-level debug --bind-address 0.0.0.0 --port 8080

Root Causes

#	Root cause	Required fix
1	Port bound to 127.0.0.1:8080; sandbox bridge can't reach loopback	-p 0.0.0.0:8080:8080
2	Gateway image runs as UID 1000:1000; without keep-id, maps to host UID 100999, not socket owner	--userns=keep-id
3	SELinux denies container_t write on user_tmp_t Podman socket	--security-opt label=disable
4	After image UID change, existing volume data owned by 100999; keep-id then can't read it	podman unshare chown -R 0:0 <volume_data> (upgrade path)
5	No sandbox JWT keys → gateway can't issue tokens to sandbox supervisor	Generate Ed25519 key pair + mount + TOML [openshell.gateway.gateway_jwt]
6	No user auth configured → CLI calls rejected	[openshell.gateway.auth] allow_unauthenticated_users = true
7	Gateway writes sandbox token files to a container-internal path, then passes that same path to the Podman API (which runs on the host) — host doesn't have the path	State dir must use a mirrored bind mount so host path == container path; set via XDG_STATE_HOME
8	Subuid/subgid warning fires falsely — check reads /etc/subuid inside the container, not from the host	Warning should be suppressed or check should use podman system info output (host-side)

Environment

• OS: Fedora 44
• Podman: rootless, SELinux enforcing
• OpenShell: 0.0.59
• Gateway deployment: containerized (podman run ghcr.io/nvidia/openshell/gateway:latest)

Logs

Gateway log (original failure — sandbox connection refused):

WARN openshell_server::compute: Sandbox failed to become ready sandbox_name=national-mammal reason=ContainerExited Container exited with code 1

Sandbox log (original failure):

Error: × Policy fetch failed after 5 attempts: failed to connect to OpenShell server

SELinux AVC denial (during debugging):

AVC avc: denied { write } for comm="openshell-gatew" name="podman.sock" dev="tmpfs"
  scontext=system_u:system_r:container_t:s0:c487,c938
  tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=sock_file permissive=0

State path error (during debugging):

Error: × status: Internal, message: "create sandbox failed: podman API error (500):
  statfs /run/user/1000/openshell-state/openshell/podman-sandbox-tokens/.../sandbox.jwt:
  no such file or directory"