Skip to content

Use cuDNN for Qwen3.8-27B long-context training - #3

Merged
zhouaihui merged 1 commit into
mainfrom
fix/qwen27b-cudnn-cp2
Nov 6, 2026
Merged

zhouaihui merged 1 commit into
mainfrom
fix/qwen27b-cudnn-cp2

Conversation

@zhouaihui

Copy link
Copy Markdown
Collaborator

Outcome

Use native cuDNN THD attention with CP2 for the Qwen3.8-27B long-context recipe. The capacity audit checks that the observed backend matches the saved configuration while retaining the full 261,888-token requirements. Allow cold TP engine compilation during initialization.

Verification

  • 103 relevant tests passed; lint, documentation links and wheel/sdist audits passed in an isolated Linux environment.
  • Eight-GPU SM90 acceptance passed: 261,888-token synthetic inputs, training 0→4, exact resume 4→8, evaluation, native export and strict loading. Both phases completed healthy with cuDNN CP2 and no fallback.
  • Acceptance uses accumulation 1; the training recipe defaults to 2. This establishes bounded capacity and workflow behavior, not quality or comparative performance.

Compatibility and provenance

The launcher ID remains qwen27b-long. Direct YAML users should use configs/recipes/qwen3_8_27b_256k_cudnn_cp2_nvlink.yaml. Explicit FA4 configurations remain supported by the audit. Checkpoint and export formats are unchanged; existing AngelSpec attribution is retained.

Select native cuDNN THD with CP2 and allow cold TP engine initialization. Validate observed attention against the saved configuration, rejecting backend mismatches while retaining full-context capacity requirements.
@zhouaihui
zhouaihui merged commit a971cd2 into main Nov 6, 2026
2 checks passed
@zhouaihui
zhouaihui deleted the fix/qwen27b-cudnn-cp2 branch November 6, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant