Skip to content

fix(ci): fail closed in community GPU workflow - #1255

Draft
JCalafato wants to merge 1 commit into
NVIDIA:mainfrom
JCalafato:fix/community-gpu-result-status
Draft

JCalafato wants to merge 1 commit into
NVIDIA:mainfrom
JCalafato:fix/community-gpu-result-status

Conversation

@JCalafato

@JCalafato JCalafato commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Background

Community GPU CI is rejected before jobs start because its shell script interpolates failure() in an unsupported expression context. Review also found that impact classification checked out and executed pull-request code on the trusted orchestration runner, mixed diagnostic text with JSON, and could report incomplete tests as successful.

This isolated repair addresses workflow syntax, trusted classification, status reporting and cleanup. It deliberately does not claim to finish the separate GPU execution migration.

Exit Criteria

  • Validate workflow expressions and execute only the captured trusted base classifier on the orchestration runner.
  • Publish success only after explicit test success and final provision-job success; fail closed for failure, cancellation, skips or missing results.
  • Keep cleanup eligible when reservation fails after allocating an instance.
  • Preserve the CPU prerequisite, family-name validation and environment protections.
  • Before ready/merge: resolve the GPU execution blockers below and pass required exact-head checks. This PR remains draft; no GPU-execution readiness is claimed.

Implementation

  • Replace the invalid status-function interpolation with supported job/step contexts passed through environment variables. Terminal publication checks the final job result as well as the recorded test result.
  • Check out the immutable captured base SHA. Invoke its existing tools.test_impact CLI with base/head diff operands, without checking out or importing PR code. Parse its pure JSON output separately from action outputs. New families absent from the trusted inventory conservatively select all trusted families.
  • Publish the deterministic instance name before reservation so timeout/failure does not suppress best-effort cleanup.
  • Add behavioral regressions for poisoned PR imports, scope classification, missing/skipped/cancelled outcomes, late job failures and reservation cleanup.
  • No model code, public API, ABI, bundle format, runtime dependencies or family quality criteria change.

Change categories

  • Model or runtime behavior
  • Public API
  • ABI
  • Bundle or artifact format
  • Dependencies
  • Documentation only
  • CI or developer tooling

Validation

Commands and Results

  • "$TEST_PYTHON" -m pytest -q tools/tests/test_community_ci.py tools/tests/test_family_impact.py tools/tests/test_architecture.py: 101 passed.
  • /tmp/trtmc-actionlint-1.7.12/actionlint -shellcheck= -pyflakes=: passed for all workflows. This validates workflow expressions/schema; optional ShellCheck/Pyflakes integrations were disabled because they are not installed.
  • Independent tools/tests/test_community_ci.py run: 36 passed, including actual shell-step execution with fake service clients and isolated Git fixtures.
  • Ruff checks/formatting and git diff --check: passed.
  • Public Community CPU run34541066607: passed on exact head f6ec6b3aa8fad4cd973508097d08fbf7daaa040e, including source quality, ownership, docs, hardened units and Required aggregate.
  • Required protected premerge: pending, dispatched after public CPU success; no protected pass is claimed. No Community GPU instance was reserved or GPU inference executed during local workflow validation.

Hardware, Environment, and Revisions

  • Candidate head: f6ec6b3aa8fad4cd973508097d08fbf7daaa040e; base: a50cf5dc215e84c1d9341a8ef72f7df746bde894.
  • Local x86_64 Linux, Python 3.12; no CUDA/TensorRT/GPU requirement for these workflow regression tests.
  • Actionlint v1.7.12 Linux amd64 archive verified against the official release checksum manifest. It is a validation tool, not a new project runtime dependency.

Not Run / Remaining Gaps

The existing GPU runner still needs a faithful family-owned E2E setup:

  • Replace obsolete tests/e2e/models/{family} paths with the existing family-owned tests.
  • Build the native executable, backend/family libraries and test targets; a py-only install is insufficient.
  • Supply explicit testcase/model selection and preserve required execution checks. Merely replacing paths can produce an all-skipped false success.
  • Stage exact checkpoint revisions in the untrusted GPU execution environment; some family tests require cached checkpoints.
  • Resolve hardware capacity without narrowing acceptance criteria: the existing shared-change smoke set includes Qwen3.8-27B FP16, whose weights alone exceed the selected single L40 capacity. Changed-family requests may also require multiple GPUs.
  • The workflow still lacks a repository-wide concurrent-resource/cost cap across distinct PRs.

No smoke-family substitution, large-test filtering, quantization workaround or threshold reduction is included. Syntax/contract validation is not evidence of a successful GPU smoke run.

Contributor Self-Review

  • I have completed a self-review of this change.

Independent review approved the bounded syntax, trust, status and cleanup repair. Actual GPU execution readiness remains unresolved and explicitly outside the validation claim.

Notes For Future Readers

Review the trusted checkout/classifier first, then fail-closed status publication and cleanup, followed by the poison-head/outcome regressions. Do not reintroduce a PR-head checkout on the trusted runner or treat missing test output as success.

This repair is separate from the Edge adapter feature in #1253. Its workflow must enter trusted main through normal reviewed merge rules before it can affect the default-branch dispatcher; no direct-main push or check bypass is intended. Keep this PR draft until runner setup and resource policy are resolved.

Risk level

  • Low
  • Medium
  • High

The diff is bounded and improves the trust boundary, but it changes orchestration that can allocate paid GPU resources. Local tests do not validate the external service or real model execution.

Use legal status contexts and require explicit test and final job success. Classify the diff with the immutable trusted base instead of executing PR Python. Preserve cleanup after partial provisioning failures and cover poisoned heads, status outcomes, and cleanup behavior.

This bounded repair does not claim the existing GPU runner migration or hardware provisioning is complete.

Signed-off-by: Joshua Calafato <jcalafato@nvidia.com>
@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@JCalafato JCalafato self-assigned this Sep 10, 2026
chaofengw-nv added a commit to chaofengw-nv/TensorRT-Model-Connect-fork that referenced this pull request Sep 11, 2026
Run both stages in one ordered workflow so GPU provisioning begins only after the CPU aggregate passes.

Carry forward trusted-base impact classification, fail-closed status publication, exact-merge validation, and deterministic Brev cleanup.

Refs: NVIDIA#1255, NVIDIA#1261
Signed-off-by: chaofengw <chaofengw@nvidia.com>
chaofengw-nv added a commit to chaofengw-nv/TensorRT-Model-Connect-fork that referenced this pull request Sep 11, 2026
Run both stages in one ordered workflow so GPU provisioning begins only after the CPU aggregate passes.

Carry forward trusted-base impact classification, fail-closed status publication, exact-merge validation, and deterministic Brev cleanup.

Refs: NVIDIA#1255, NVIDIA#1261
Signed-off-by: chaofengw <chaofengw@nvidia.com>
chaofengw-nv added a commit to chaofengw-nv/TensorRT-Model-Connect-fork that referenced this pull request Sep 11, 2026
Introduce a single exact-merge pipeline that authorizes GPU work only after the CPU aggregate succeeds. Preserve the CPU-gated run-internal-ci label bridge and retain the standalone CPU workflow only for rollout compatibility.

Run explicit family-owned premerge E2E with native artifacts, staged checkpoints, fail-closed status and cleanup, and repository-wide GPU serialization.

Refs: NVIDIA#1255

Refs: NVIDIA#1261
Signed-off-by: chaofengw <chaofengw@nvidia.com>
chaofengw-nv added a commit to chaofengw-nv/TensorRT-Model-Connect-fork that referenced this pull request Sep 11, 2026
Introduce a single exact-merge pipeline that authorizes GPU work only after the CPU aggregate succeeds. Preserve the CPU-gated run-internal-ci label bridge and retain the standalone CPU workflow only for rollout compatibility.

Run explicit family-owned premerge E2E with native artifacts, staged checkpoints, fail-closed status and cleanup, and repository-wide GPU serialization.

Keep the hardened CPU image capable of executing embedded workflow shell regressions, including their jq dependency.

Refs: NVIDIA#1255

Refs: NVIDIA#1261
Signed-off-by: chaofengw <chaofengw@nvidia.com>
chaofengw-nv added a commit to chaofengw-nv/TensorRT-Model-Connect-fork that referenced this pull request Sep 11, 2026
Introduce a single exact-merge pipeline that authorizes GPU work only after the CPU aggregate succeeds. Preserve the CPU-gated run-internal-ci label bridge and retain the standalone CPU workflow only for rollout compatibility.

Run explicit family-owned premerge E2E with native artifacts, staged checkpoints, fail-closed status and cleanup, and repository-wide GPU serialization.

Keep the hardened CPU image capable of executing embedded workflow shell regressions, including their jq dependency. Carry the live Brev Docker permission fix by using sudo on freshly reserved instances.

Refs: NVIDIA#1255

Refs: NVIDIA#1261

Refs: NVIDIA#1264
Signed-off-by: chaofengw <chaofengw@nvidia.com>
chaofengw-nv added a commit to chaofengw-nv/TensorRT-Model-Connect-fork that referenced this pull request Sep 11, 2026
Introduce a single exact-merge pipeline that authorizes GPU work only after the CPU aggregate succeeds. Preserve the CPU-gated run-internal-ci label bridge and retain the standalone CPU workflow only for rollout compatibility.

Run explicit family-owned premerge E2E with native artifacts, staged checkpoints, fail-closed status and cleanup, and repository-wide GPU serialization.

Keep the hardened CPU image capable of executing embedded workflow shell regressions, including their jq dependency. Carry the live Brev Docker permission fix by using sudo on freshly reserved instances.

Refs: NVIDIA#1255

Refs: NVIDIA#1261

Refs: NVIDIA#1264
Signed-off-by: chaofengw <chaofengw@nvidia.com>
chaofengw-nv added a commit to chaofengw-nv/TensorRT-Model-Connect-fork that referenced this pull request Sep 11, 2026
Introduce a single exact-merge pipeline that authorizes GPU work only after the CPU aggregate succeeds. Preserve the CPU-gated run-internal-ci label bridge and retain the standalone CPU workflow only for rollout compatibility.

Run explicit family-owned premerge E2E with native artifacts, staged checkpoints, fail-closed status and cleanup, and repository-wide GPU serialization.

Keep both public and protected hardened CPU images capable of executing embedded workflow shell regressions, including their jq dependency. Carry the live Brev Docker permission fix by using sudo on freshly reserved instances.

Refs: NVIDIA#1255

Refs: NVIDIA#1261

Refs: NVIDIA#1264
Signed-off-by: chaofengw <chaofengw@nvidia.com>
chaofengw-nv added a commit that referenced this pull request Sep 11, 2026
Introduce a single exact-merge pipeline that authorizes GPU work only after the CPU aggregate succeeds. Preserve the CPU-gated run-internal-ci label bridge and retain the standalone CPU workflow only for rollout compatibility.

Run explicit family-owned premerge E2E with native artifacts, staged checkpoints, fail-closed status and cleanup, and repository-wide GPU serialization.

Keep both public and protected hardened CPU images capable of executing embedded workflow shell regressions, including their jq dependency. Carry the live Brev Docker permission fix by using sudo on freshly reserved instances.

Refs: #1255

Refs: #1261

Refs: #1264

Signed-off-by: chaofengw <chaofengw@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant