From e175ea3225c69d92fa83b09ea0337c45e7cc099f Mon Sep 17 00:00:00 2001
From: Zhenshan Xie <zhenshanx@nvidia.com>
Date: Fri, 11 Sep 2026 00:40:21 -0700
Subject: [PATCH] fix(ci): use sudo for docker on freshly reserved GPU
 instances

The second real live trigger of community-gpu-ci.yml (PR #1260, after
fixing the failure() expression and impact-JSON bugs in #1252/#1261)
reached provision-and-test and actually reserved a GPU, but failed with:

  ERROR: permission denied while trying to connect to the docker API
  at unix:///var/run/docker.sock

right on the first docker command after instance creation. git clone
succeeded on the same instance immediately before it, so SSH access
itself was fine; the instance's docker-group membership for the SSH
session had not propagated yet. Our earlier manual proof-of-concept
never hit this because there was always a natural delay (multiple
separate exec calls, manual inspection) between instance creation and
the first docker command; this workflow goes from reserve straight
into docker build with no gap.

Fix: prefix both docker invocations (build and run) with sudo, which
sidesteps the group-membership timing question entirely instead of
depending on it. Confirmed teardown already worked correctly on the
failed run (brev delete ran via the always() step, no orphaned
instance), so this is the last known blocker from the two prior
live-fire attempts.

Signed-off-by: Zhenshan Xie <zhenshanx@nvidia.com>
---
 .github/workflows/community-gpu-ci.yml | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/community-gpu-ci.yml b/.github/workflows/community-gpu-ci.yml
index 29e2cefa8..078d864d6 100644
--- a/.github/workflows/community-gpu-ci.yml
+++ b/.github/workflows/community-gpu-ci.yml
@@ -263,7 +263,13 @@ jobs:
         run: |
           set -euo pipefail
           brev exec "$INSTANCE_NAME" "git clone --no-checkout https://github.com/$GITHUB_REPOSITORY.git /tmp/model_connect && cd /tmp/model_connect && git fetch --depth 1 origin $HEAD_SHA && git checkout $HEAD_SHA"
-          brev exec "$INSTANCE_NAME" "cd /tmp/model_connect && docker build -f Dockerfile.dev.x86-gpu -t trtmc-quickstart-gpu requirements"
+          # sudo: a freshly created instance's SSH session does not
+          # reliably have its docker-group membership propagated yet
+          # (confirmed live: "permission denied ... docker.sock" on the
+          # very first docker command right after instance creation).
+          # sudo sidesteps the group-membership timing entirely rather
+          # than depending on it.
+          brev exec "$INSTANCE_NAME" "cd /tmp/model_connect && sudo docker build -f Dockerfile.dev.x86-gpu -t trtmc-quickstart-gpu requirements"
 
           # Selective by design: 87 model families exist, and running the
           # full tests/e2e/models suite on GPU for every push is not
@@ -297,7 +303,7 @@ jobs:
             exit 0
           fi
 
-          brev exec "$INSTANCE_NAME" "docker run --rm --gpus all -v /tmp/model_connect:/src -w /src trtmc-quickstart-gpu bash -c 'python3.12 -m pip install --no-deps -e . -C py-only=true && python3.12 -m pytest $test_paths -v'" \
+          brev exec "$INSTANCE_NAME" "sudo docker run --rm --gpus all -v /tmp/model_connect:/src -w /src trtmc-quickstart-gpu bash -c 'python3.12 -m pip install --no-deps -e . -C py-only=true && python3.12 -m pytest $test_paths -v'" \
             | tee /tmp/gpu-ci-output.log
           echo "conclusion=success" >> "$GITHUB_OUTPUT"