Skip to content

fix: handle empty one-dimensional tensors in concatenation - #4701

Open
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:upstream/cat-rank1-empty-operand
Open

shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:upstream/cat-rank1-empty-operand

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Problem

Joining an empty tensor with a larger tensor can force a model to run that operation in PyTorch instead of TensorRT. This happens when a model starts with an empty cache and adds values to it.

PyTorch accepts an empty tensor with shape (0,) alongside tensors with more dimensions. TensorRT requires matching numbers of dimensions.

Change

Skip these empty tensors when checking dimensions and building the TensorRT concatenation. Continue rejecting dimension mismatches between the remaining tensors.

An empty tensor can still affect the result's data type. Choose the output type using every input before removing empty ones. Otherwise, later arithmetic could overflow in float16 when ordinary PyTorch would use float32.

The validator's test inputs include the data type it needs.

Tests

Passed 10/10 focused tests: four validator tests and six tests that build and run TensorRT engines. They cover empty tensors in different positions, positive and negative axes, mixed input types, and a float64 constant with conversion to float32 enabled. The empty-input validator regression fails on the base branch.

After rebasing, the existing nonzero suite passed 17 cases and skipped its RTX-only case, checking the nearby upstream converter changes.

The focused rerun used Linux x86_64, Python 3.12, PyTorch 2.15 nightly, and TensorRT 11.3 with the Python runtime. Earlier native-runtime coverage used TensorRT 11.2; a native TensorRT 11.3 build was not tested.

Windows, aarch64, TensorRT-RTX, TensorRT 10.x, and concatenation dimensions that vary between calls were not tested.

@meta-cla meta-cla Bot added the cla signed label Sep 10, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Sep 10, 2026
## Problem

Joining an empty tensor with a larger tensor can force a model to run that operation in PyTorch instead of TensorRT. This happens when a model starts with an empty cache and adds values to it.

PyTorch accepts an empty tensor with shape `(0,)` alongside tensors with more dimensions. TensorRT requires matching numbers of dimensions.

## Change

Skip these empty tensors when checking dimensions and building the TensorRT concatenation. Continue rejecting dimension mismatches between the remaining tensors.

An empty tensor can still affect the result's data type. Choose the output type using every input before removing empty ones. Otherwise, later arithmetic could overflow in float16 when ordinary PyTorch would use float32.

The validator's test inputs now include the data type it needs.

## Tests

Passed 10/10 focused tests: four validator tests and six tests that build and run TensorRT engines. They cover empty tensors in different positions, positive and negative axes, and mixed input types. They also cover a float64 constant with conversion to float32 enabled.

Tested on Linux x86_64 with Python 3.12, TensorRT 11.2, and the native runtime. Windows, aarch64, TensorRT-RTX, TensorRT 10.x, and input dimensions that vary between calls were not tested.
@shoumikhin
shoumikhin force-pushed the upstream/cat-rank1-empty-operand branch from bef9bca to a795cbd Compare September 12, 2026 04:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant