Skip to content

Fix privileged AWF environment forwarding on sudo-rs - #72409

Merged
lpcox merged 1 commit into
mainfrom
pelikhan/ideal-spoon
Oct 27, 2026
Merged

lpcox merged 1 commit into
mainfrom
pelikhan/ideal-spoon

Conversation

@pelikhan

Copy link
Copy Markdown
Collaborator

Why

Daily MCP Tool Concurrency Analysis run 40480413637 failed before agent work started. sudo-rs ignored whole-environment --preserve-env, stripping runner identity and credentials. AWF then misclassified the host and fell back from Cloud Hypervisor to Docker, where the guest /workspace write policy failed validation. The later resolved-config EEXIST warning was secondary retry finalization fallout.

Approach

  • Preserve exported environment variables explicitly by name for privileged Cloud Hypervisor, NVX, and legacy AWF launchers. Values remain in the environment rather than being serialized into command arguments.
  • Restore runtime PATH through /usr/bin/env while executing the fixed /usr/local/bin/awf entrypoint. Rootless defaults, custom command precedence, credential exclusions, filesystem policy, firewall, and attestation remain unchanged.
  • Add a mocked sudo-rs regression that reproduces the previous environment loss and checks runner metadata, credentials, whitespace, multiline values, and shell metacharacters.
  • Regenerate the eight affected workflow locks and document environment forwarding. No workflow Markdown was changed.

Validation

  • Targeted launcher/profile regressions, impacted unit tests, build, formatting, schema freshness, custom linters, and Go lint passed.
  • Native actionlint/shellcheck and normal compilation passed; all 349 workflow locks are in sync.
  • Source and generated-lock security review passed with no confirmed vulnerabilities.
  • The aggregate gate encountered a concurrent golangci-lint lock; the same lint completed successfully with serial-runner support.

Remaining uncertainty

No workflows were dispatched or rerun. Hosted recovery is therefore unverified. Diagnostic dry-run compilation failed closed because the existing workflow uses mcp-scripts, which is not supported in dry-run; no diagnostic lock was emitted and no protections were removed.

Related failure report: #72370.

Pass exported variable names explicitly to sudo and retain trusted system entrypoints. This prevents Cloud Hypervisor host misclassification and Docker fallback after sudo-rs drops the launcher environment.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator Author

@lpcox

@pelikhan
pelikhan requested a review from lpcox October 27, 2026 11:01
@pelikhan pelikhan added the awf label Oct 27, 2026
@pelikhan
pelikhan marked this pull request as ready for review October 27, 2026 15:44
Copilot AI max review requested due to automatic review settings October 27, 2026 15:44
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus55 · ~12 AIC · ⌖ ~38 AIC · ◇ ~4 AIC · ⊞ 1.9K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · haiku45 · ~6 AIC · ⌖ ~9 AIC · ◇ ~3 AIC · ⊞ 8.9K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · ~10 AIC · ⌖ ~7 AIC · ⊞ 14.8K · ◷

Generated by Ponytail Reviewer for #72409

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-10-27T15:49:17Z
review_event: REQUEST_CHANGES
top_themes:
  - privileged environment forwarding exposes the full exported environment to root AWF
files_reviewed:
  - pkg/constants/constants.go
  - pkg/workflow/awf_command_builder.go
  - pkg/workflow/awf_command_builder_test.go
  - docs/src/content/docs/reference/sandbox.md
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes

The new privileged launcher forwards the entire exported shell environment to a root AWF process. This creates a materially broader privilege boundary than the prior behavior and can carry loader/interpreter controls into privileged execution. Restrict preservation to an explicit safe allowlist and add coverage proving unsafe variables are excluded.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

// whole-environment preservation (-E). Only names, never credential values, enter
// sudo's arguments. Restore PATH for host tools but execute the trusted system AWF.
const AWFPrivilegedCommand = `sudo --preserve-env="$(compgen -e | paste -sd, -)" /usr/bin/env PATH="$PATH" /usr/local/bin/awf`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This forwards every exported variable from the workflow shell into a root process, widening the privileged environment boundary and allowing inherited loader/interpreter controls such as LD_PRELOAD, BASH_ENV, ENV, NODE_OPTIONS, or PYTHONPATH to reach /usr/local/bin/awf; that can alter privileged execution and defeats the security rationale for replacing sudo -E.

💡 Preserve an explicit safe allowlist

Build the --preserve-env list from only the runner/AWF variables required for identity and operation, and explicitly exclude loader/interpreter injection variables. Add a regression assertion that unsafe variables are not forwarded.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd (per pr-triage) — no blocking issues found.

📋 Analysis

Root cause & fix

The diagnosis is sound: sudo-rs's -E/bare --preserve-env silently drops the whole environment instead of erroring, which is exactly the kind of failure mode that's easy to miss without an explicit regression. The fix in pkg/constants/constants.go (naming variables via compgen -e | paste -sd, - rather than relying on whole-environment preservation) correctly addresses the root cause rather than papering over the symptom, and keeps credential values out of sudo's argv (only names are interpolated).

Regression test quality (/tdd)

TestPrivilegedAWFEnvironmentPreservation in awf_command_builder_test.go is a strong regression test: it mocks sudo-rs's actual broken behavior (ignoring bare --preserve-env, applying secure_path), and asserts the old command genuinely reproduces data loss (runner identity, COPILOT_GITHUB_TOKEN, whitespace, multiline values, shell metacharacters) while the new command preserves it. That's a textbook "prove the bug, then prove the fix" regression test rather than just a snapshot assertion.

Verified independently

  • Confirmed compgen -e is a bash builtin and all AWF invocation call sites run under bash//bin/bash -c, so the new command is portable within this codebase's shell usage.
  • Confirmed /usr/local/bin/awf matches the non-rootless installer path (existing test already enforces this).
  • Verified the generated lock diffs are a pure 1-line mechanical regeneration consistent with the constant change — no drift.

Minor, non-blocking observation (/codebase-design)

AWFCloudHypervisorCommand, AWFNVXCommand, and AWFLegacySecurityCommand are now textually identical aliases of AWFPrivilegedCommand. That's a reasonable short-term tradeoff to avoid touching every call site/test reference, but if a future change needs runtime-specific sudo behavior (e.g., different --exclude-env semantics per launcher), these three names could silently diverge from their doc comments. Not blocking — just worth a TODO if the three ever need to differ.

Positive highlights

  • ✅ Root-caused via upstream failure report (#72370) rather than guessing
  • ✅ Documentation updated in docs/src/content/docs/reference/sandbox.md to clarify credential exclusions still apply
  • ✅ No workflow Markdown touched — lock regeneration is purely mechanical

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · ~104 AIC · ⌖ ~18 AIC · ⊞ 10.2K
Comment /matt to run again

@lpcox
lpcox merged commit 0627fdb into main Oct 27, 2026
471 of 472 checks passed
@lpcox
lpcox deleted the pelikhan/ideal-spoon branch October 27, 2026 16:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

7 open issues
What changed in this PR

This PR standardizes privileged AWF invocation to work reliably on sudo-rs hosts by explicitly preserving exported environment variables (by name) and restoring PATH, then updates tests, docs, and workflow lockfiles to match.

Changes:

  • Introduces a shared privileged AWF command prefix and reuses it for cloud-hypervisor, NVX, and legacy-security paths.
  • Refactors multiple workflow/unit tests to assert against constants instead of hard-coded sudo strings, and adds a new env preservation behavior test.
  • Updates documentation and workflow lockfiles to reflect the new privileged invocation format.
File Description
pkg/​workflow/​sandbox_network_isolation_rootless_test.go Uses constants for legacy security sudo invocation assertion
pkg/​workflow/​firewall_workflow_test.go Uses constants for legacy security sudo invocation assertion
pkg/​workflow/​enclave_cloud_hypervisor_test.go Uses constants for cloud-hypervisor AWF command prefix/assertions
pkg/​workflow/​cloud_hypervisor_test.go Uses constants for cloud-hypervisor AWF invocation assertions
pkg/​workflow/​awf_command_builder_test.go Adds test validating privileged env preservation behavior (sudo-rs model)
pkg/​workflow/​awf_command_builder.go Updates GetAWFCommandPrefix docstring to match new behavior
pkg/​constants/​constants.go Adds AWFPrivilegedCommand and aliases multiple command constants to it
docs/​src/​content/​docs/​reference/​sandbox.md Documents privileged profiles preserving runner env variables by name
.github/​workflows/​smoke-work-queue.lock.yml Updates locked workflow to use the new privileged sudo invocation
.github/​workflows/​smoke-service-ports.lock.yml Updates locked workflow to use the new privileged sudo invocation
.github/​workflows/​daily-mcp-concurrency-analysis.lock.yml Updates locked workflow to use the new privileged sudo invocation
.github/​workflows/​daily-fact.lock.yml Updates locked workflow to use the new privileged sudo invocation
.github/​workflows/​daily-caveman-optimizer.lock.yml Updates locked workflow to use the new privileged sudo invocation
.github/​workflows/​daily-awf-spec-compiler-surfacing.lock.yml Updates locked workflow to use the new privileged sudo invocation
.github/​workflows/​copilot-centralization-optimizer.lock.yml Updates locked workflow to use the new privileged sudo invocation
.github/​workflows/​audit-workflows.lock.yml Updates locked workflow to use the new privileged sudo invocation

Review effort: Lite

Comment on lines +240 to +243
// AWFPrivilegedCommand names exported variables explicitly because sudo-rs ignores
// whole-environment preservation (-E). Only names, never credential values, enter
// sudo's arguments. Restore PATH for host tools but execute the trusted system AWF.
const AWFPrivilegedCommand = `sudo --preserve-env="$(compgen -e | paste -sd, -)" /usr/bin/env PATH="$PATH" /usr/local/bin/awf`
Comment on lines +248 to +251
const AWFCloudHypervisorCommand = AWFPrivilegedCommand

// AWFNVXCommand runs AWF with the host privileges required to create NVX microVMs.
const AWFNVXCommand = "sudo --preserve-env awf"
const AWFNVXCommand = AWFPrivilegedCommand
// AWFLegacySecurityCommand runs the trusted system AWF binary with the runner's
// PATH restored after sudo applies secure_path.
const AWFLegacySecurityCommand = `sudo -E /usr/bin/env PATH="$PATH" /usr/local/bin/awf`
const AWFLegacySecurityCommand = AWFPrivilegedCommand

Omitting `runtime` is equivalent to `runtime: docker`, which keeps the secure default.

Privileged profiles invoke the system-installed AWF binary and preserve exported runner variables by name, including on `sudo-rs` hosts that ignore `sudo -E`. AWF's agent credential exclusions still apply; preserving the launcher environment does not grant those credentials to the agent.
// AWFPrivilegedCommand names exported variables explicitly because sudo-rs ignores
// whole-environment preservation (-E). Only names, never credential values, enter
// sudo's arguments. Restore PATH for host tools but execute the trusted system AWF.
const AWFPrivilegedCommand = `sudo --preserve-env="$(compgen -e | paste -sd, -)" /usr/bin/env PATH="$PATH" /usr/local/bin/awf`
"synthetic-test-token", custom, path, "",
}, strings.Split(string(output), "\x00"))

oldScript := strings.Replace(script, prefix, `sudo --preserve-env /usr/bin/env PATH="$PATH" /usr/local/bin/awf`, 1)
[[ "$names" == *,RUNNER_TEMP,* ]] && preserved+=("RUNNER_TEMP=$RUNNER_TEMP")
[[ "$names" == *,COPILOT_GITHUB_TOKEN,* ]] && preserved+=("COPILOT_GITHUB_TOKEN=$COPILOT_GITHUB_TOKEN")
[[ "$names" == *,GH_AW_TEST_CUSTOM_ENV,* ]] && preserved+=("GH_AW_TEST_CUSTOM_ENV=$GH_AW_TEST_CUSTOM_ENV")
[[ "$names" != *"$COPILOT_GITHUB_TOKEN"* ]]
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.0.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants