Add weightless hyper-connection fusion operators

[Copilot]

Description

Adds four com.microsoft operators for weightless hyper-connections:

• BranchwiseRMSNorm
• ScaledSiLU
• HyperConnectionPreMix
• HyperConnectionPostMix

Key changes:

• Adds schemas, CPU/CUDA/WebGPU registrations, and per-operator source files.
• Reuses existing SimplifiedLayerNormalization implementations for compatible BranchwiseRMSNorm paths.
• Retains targeted fallbacks for optional, mixed-type, and branch-specific scales.
• Adds dedicated symbolic shape-inference handlers for each operator.
• Adds operator coverage and generated documentation.

Motivation and Context

Provides portable fused primitives for weightless hyper-connection models while sharing established normalization infrastructure where semantics permit. Dedicated schemas and shape-inference handlers preserve each operator’s distinct contract without coupling them to unrelated operators.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

The flattened symbolic-shape test relies on an expression-like dimension name that the inference engine treats atomically, and important normalization paths remain untested.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds four fused hyper-connection contrib operators across CPU, CUDA, and WebGPU, including schemas, shape inference, tests, and generated documentation.

Changes:

• Implements BranchwiseRMSNorm, ScaledSiLU, PreMix, and PostMix kernels.
• Adds shared validation and symbolic shape inference.
• Registers and documents the operators across supported providers.

File summaries

File	Description
onnxruntime/test/python/onnxruntime_test_python_symbolic_shape_infer.py	Adds symbolic shape tests.
onnxruntime/test/contrib_ops/hyper_connection_op_test.cc	Adds operator tests.
onnxruntime/python/tools/symbolic_shape_infer.py	Adds symbolic inference handlers.
onnxruntime/core/providers/cpu/nn/layer_norm_impl.h	Exposes parameterized normalization construction.
onnxruntime/core/providers/cpu/nn/layer_norm_impl.cc	Implements reusable normalization setup.
onnxruntime/core/graph/contrib_ops/ms_opset.h	Registers operator schemas.
onnxruntime/core/graph/contrib_ops/bert_defs.cc	Defines schemas and shape inference.
onnxruntime/contrib_ops/hyper_connection_helper.h	Provides shared shape validation.
onnxruntime/contrib_ops/cpu/cpu_contrib_kernels.cc	Registers CPU kernels.
onnxruntime/contrib_ops/cpu/bert/scaled_silu.h	Declares CPU ScaledSiLU.
onnxruntime/contrib_ops/cpu/bert/scaled_silu.cc	Implements CPU ScaledSiLU.
onnxruntime/contrib_ops/cpu/bert/hyper_connection_pre_mix.h	Declares CPU PreMix.
onnxruntime/contrib_ops/cpu/bert/hyper_connection_pre_mix.cc	Implements CPU PreMix.
onnxruntime/contrib_ops/cpu/bert/hyper_connection_post_mix.h	Declares CPU PostMix.
onnxruntime/contrib_ops/cpu/bert/hyper_connection_post_mix.cc	Implements CPU PostMix.
onnxruntime/contrib_ops/cpu/bert/branchwise_rms_norm.h	Declares CPU branchwise RMS normalization.
onnxruntime/contrib_ops/cpu/bert/branchwise_rms_norm.cc	Implements CPU branchwise RMS normalization.
onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc	Registers CUDA kernels.
onnxruntime/contrib_ops/cuda/bert/scaled_silu.h	Declares CUDA ScaledSiLU.
onnxruntime/contrib_ops/cuda/bert/scaled_silu.cc	Integrates CUDA ScaledSiLU kernel.
onnxruntime/contrib_ops/cuda/bert/scaled_silu_impl.h	Declares CUDA launcher.
onnxruntime/contrib_ops/cuda/bert/scaled_silu_impl.cu	Implements CUDA ScaledSiLU.
onnxruntime/contrib_ops/cuda/bert/hyper_connection_pre_mix.h	Declares CUDA PreMix.
onnxruntime/contrib_ops/cuda/bert/hyper_connection_pre_mix.cc	Integrates CUDA PreMix kernel.
onnxruntime/contrib_ops/cuda/bert/hyper_connection_pre_mix_impl.h	Declares CUDA PreMix launcher.
onnxruntime/contrib_ops/cuda/bert/hyper_connection_pre_mix_impl.cu	Implements CUDA PreMix.
onnxruntime/contrib_ops/cuda/bert/hyper_connection_post_mix.h	Declares CUDA PostMix.
onnxruntime/contrib_ops/cuda/bert/hyper_connection_post_mix.cc	Integrates CUDA PostMix kernel.
onnxruntime/contrib_ops/cuda/bert/hyper_connection_post_mix_impl.h	Declares CUDA PostMix launcher.
onnxruntime/contrib_ops/cuda/bert/hyper_connection_post_mix_impl.cu	Implements CUDA PostMix.
onnxruntime/contrib_ops/cuda/bert/branchwise_rms_norm.h	Declares CUDA branchwise RMS normalization.
onnxruntime/contrib_ops/cuda/bert/branchwise_rms_norm.cc	Integrates CUDA normalization paths.
onnxruntime/contrib_ops/cuda/bert/branchwise_rms_norm_impl.h	Declares mixed-scale CUDA launcher.
onnxruntime/contrib_ops/cuda/bert/branchwise_rms_norm_impl.cu	Implements mixed-scale CUDA normalization.
onnxruntime/contrib_ops/webgpu/webgpu_contrib_kernels.cc	Registers WebGPU kernels.
onnxruntime/contrib_ops/webgpu/bert/scaled_silu.h	Declares WebGPU ScaledSiLU.
onnxruntime/contrib_ops/webgpu/bert/scaled_silu.cc	Implements WebGPU ScaledSiLU.
onnxruntime/contrib_ops/webgpu/bert/hyper_connection_pre_mix.h	Declares WebGPU PreMix.
onnxruntime/contrib_ops/webgpu/bert/hyper_connection_pre_mix.cc	Implements WebGPU PreMix.
onnxruntime/contrib_ops/webgpu/bert/hyper_connection_post_mix.h	Declares WebGPU PostMix.
onnxruntime/contrib_ops/webgpu/bert/hyper_connection_post_mix.cc	Implements WebGPU PostMix.
onnxruntime/contrib_ops/webgpu/bert/branchwise_rms_norm.h	Declares WebGPU branchwise RMS normalization.
onnxruntime/contrib_ops/webgpu/bert/branchwise_rms_norm.cc	Implements WebGPU normalization paths.
docs/OperatorKernels.md	Documents provider kernel support.
docs/ContribOperators.md	Documents the new operator schemas.

Review details

• Files reviewed: 45/45 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Generated documentation is inconsistent with its schema source, and the CUDA RMS normalization kernel is severely under-parallelized.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

onnxruntime/contrib_ops/cuda/bert/branchwise_rms_norm_impl.cu:40

• The launch assigns one thread to an entire branch vector, and that thread serially traverses hidden twice. For common decode shapes such as one row with four branches, this launches only four active threads while each performs thousands of operations, making the CUDA fused op severely underutilize the GPU. Use one block per group, distribute h across threads, and reduce the sum of squares cooperatively (as the existing per-head RMS normalization kernel does).

• Files reviewed: 45/45 changed files
• Comments generated: 1
• Review effort level: Balanced

[Tianlei Wu (tianleiwu)]

How does these operator related to #32123?

[kunal-vaishnavi]

How does these operator related to #32123?

The main difference is these operators are weightless. The operator in your PR fuses the MatMul ops inside them and the hyper-connection subgraph is materially different between Qwen-3.8 Flash Next and DeepSeek V4. You would need to include a new op schema for differentiating how the op works for the two models.

On the other hand, the pre-mix and post-mix subgraphs are more generic and applicable to both models. Those subgraphs are still weightless, and their operator fusion still improves runtime performance. I think it should be fine to include these two new ops. In the future, they can be fused together with the hyper-connection subgraphs to form the HyperConnectionMix op proposed in your PR.

[Copilot]

Copilot review overview

🟡 Changes recommended

CPU and WebGPU normalization reuse paths violate the declared float32 arithmetic contract and produce inconsistent results.

Review effort: Balanced
Findings: 2

Open (2)

• Branch-specific scales bypass required float32 RMSNorm arithmetic · New
• Float16 shared-scale RMSNorm uses insufficient precision · New

Resolved since last review (1)

• This schema documentation names the operand branch_output, while the declared input and…