[TRTLLMINF-357][infra] Consolidate runBranchesWithInfraDefer into shared lib - #18385
Draft
brnguyen2 wants to merge 2 commits into
Draft
[TRTLLMINF-357][infra] Consolidate runBranchesWithInfraDefer into shared lib#18385brnguyen2 wants to merge 2 commits into
brnguyen2 wants to merge 2 commits into
Conversation
…red lib Both jenkins/L0_Test.groovy and jenkins/Build.groovy carried near-identical inline copies of the infra-scoped fail-fast helper (added by TRTLLMINF-311 and TRTLLMINF-324). Replace both with calls to the new shared trtllm_utils.runBranchesWithInfraDefer, which owns the parallel/failFast/UNSTABLE orchestration. Scope policy stays in each caller via an injected isDeferrableInfra(e, stageName) closure, since trtllm.FailureClassifier / InfraFailure cannot be imported into vars/trtllm_utils.groovy without breaking its loadScript unit tests: - Build.groovy: K8s-only (build pods are K8s builders). - L0_Test.groovy: infraDeferPredicate() checks K8S for every stage and SLURM in addition for SLURM-dispatcher stages, preserving the prior per-stage behavior. Behavior is unchanged; still gated on ENABLE_INFRA_SCOPED_FAILFAST. Requires the paired shared-lib change (trtllm_utils.runBranchesWithInfraDefer) on trtllm-jenkins-shared-lib@main. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
Throwaway commit to validate PR NVIDIA#18385 against the paired shared-lib branch (trtllm-jenkins-shared-lib@user/brnguyen/TRTLLMINF-357-shared-infradefer, MR !104) before it merges to main. MUST be reverted before this PR merges -- the merged version references trtllm-jenkins-shared-lib@main. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #70046 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #70046 [ run ] completed with state
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Both
jenkins/L0_Test.groovyandjenkins/Build.groovycarried near-identical inline copies of the infra-scoped fail-fast helperrunBranchesWithInfraDefer(added by TRTLLMINF-311 at the test layer and TRTLLMINF-324 at the build layer; the Build copy was inline only because the original lived in the L0_Test script's own scope, with aTODO(TRTLLMINF-324)to de-duplicate).This PR removes both inline copies and calls the new shared helper
trtllm_utils.runBranchesWithInfraDefer, which owns the parallel /failFast/UNSTABLEorchestration.Jira: TRTLLMINF-357.
Changes
jenkins/Build.groovy— delete the inlinerunBranchesWithInfraDefer; thestage("Build")call now uses the shared helper with a K8s-only deferral closure (build pods are K8s builders).jenkins/L0_Test.groovy— delete the inlinerunBranchesWithInfraDefer; add a smallinfraDeferPredicate(stageScopes)builder (K8S for every stage, plus SLURM for SLURM-dispatcher stages) and route all four call sites through the shared helper.Scope policy stays in each caller via an injected
isDeferrableInfra(e, stageName)closure. This is required, not just stylistic:trtllm.FailureClassifier/trtllm.exceptions.InfraFailurecannot be imported intovars/trtllm_utils.groovywithout breaking itsloadScriptunit tests (the test classpath has nosrc/trtllm), and it puts the K8s-vs-SLURM decision where the per-stage scope is known.Behavior
Unchanged. Still gated on
ENABLE_INFRA_SCOPED_FAILFAST(flag off = plainfailFast+parallel, exactly as before). Only the helper's home moves.Dependency / merge order
Requires the paired shared-lib change on
trtllm-jenkins-shared-lib@main(addstrtllm_utils.runBranchesWithInfraDefer+InfraDeferTest): MR ftp/infra/trtllm-jenkins-shared-lib!104. Both pipelines pintrtllm-jenkins-shared-lib@main, so this PR must merge after !104 lands on main. Kept as draft until then. Live-pipeline validation of the pair can be done by temporarily repointing the@Libraryref to the shared-lib branch.Test Coverage
The shared helper is unit-tested in the paired shared-lib MR (loadScript + PipelineStub: disabled pass-through, infra-defer + UNSTABLE, genuine-failure rethrow, interrupt-never-swallowed, all-branches outage warning). This PR is a pure Jenkins-pipeline refactor validated by pipeline runs.
PR Checklist