[VL] Support native Spark 4.1 OneRowRelationExec

[luis4a0]

What changes are proposed in this pull request?

Spark 4.1 uses OneRowRelationExec as the physical source for zero-column,
one-row inputs in literal-only queries and scalar subqueries. The Velox backend
does not currently transform this leaf, so an otherwise native plan must retain
a Spark operator and cross a row-to-columnar boundary.

This PR adds native Spark 4.1 OneRowRelationExec support by:

• recognizing the operator in the Spark 4.1 shim without introducing a Spark
  4.1 dependency into common code;
• replacing it only for the Velox backend while preserving the original Spark
  plan for unsupported backends and Spark versions;
• representing the logical row as a zero-column, one-row Substrait virtual
  table and converting it to a Velox ValuesNode;
• introducing an explicit, conservative no-input execution capability so a
  supported dependency-free native stage receives exactly one Spark partition;
• propagating that capability only through safe unary, union, and streamed-side
  broadcast-join paths;
• validating virtual-table schemas, row cardinality, null handling, malformed
  values, and split-free native conversion; and
• documenting the supported virtual-table and OneRowRelationExec behavior.

This is needed for Spark 4.1 native-plan completeness. The existing Spark
fallback produces correct results, but it fragments literal-only and
scalar-subquery pipelines and leaves an avoidable execution boundary. The new
no-input behavior is deliberately opt-in: unrelated empty or broadcast-only
stages retain their previous failure behavior, and unsupported backends retain
their original Spark plans.

This PR is not presented as a performance optimization. Its purpose is to close
the Spark 4.1 capability gap with explicit execution semantics and regression
coverage.

How was this patch tested?

Added and ran JVM unit, Spark integration, and native C++ tests covering:

• typed Spark 4.1 recognition and false-positive rejection;
• unsupported-backend identity preservation;
• Substrait virtual-table protobuf generation, batching, schema and null
  validation, zero-column cardinality, and legal zero-row execution;
• dependency-free and broadcast-only partition discovery, including concurrent
  access and unsupported-stage rejection;
• AQE enabled and disabled, scalar-subquery reuse, native union, broadcast hash
  join, broadcast nested-loop join, and multi-partition file-scan coexistence;
• Spark plan-tag preservation, exact output rows, native-stage presence, and
  removal of the OneRow-specific fallback boundary; and
• native ValuesNode execution, malformed and non-literal rejection,
  trivial-leaf registration, and file-split isolation.

Results on the current main base:

• 14 focused gluten-substrait tests passed.
• 3 Spark 4.1 shim tests passed.
• All 106 executed MiscOperatorSuite tests passed; one existing test was
  ignored.
• The Spark 4.1 explain regression passed.
• 52 native plan-conversion tests passed; 7 existing round-trip tests were
  skipped.
• Spotless, Scalastyle, Checkstyle, clang-format-15, license-header, and
  source-hygiene checks passed.

Was this patch authored or co-authored using generative AI tooling?

Co-authored by: GitHub Copilot CLI 1.0.83-0

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

🔵 Needs a closer look

It changes execution semantics and plan conversion across JVM + native (including partition discovery behavior), which warrants final human review despite strong test coverage.

Pull request overview

Adds Velox-backend support for Spark 4.1’s OneRowRelationExec by recognizing it in the Spark 4.1 shim, transforming it into a native Velox-friendly leaf (ValuesNode via Substrait ReadRel.VirtualTable), and introducing an explicit “no-input execution” capability to safely run dependency-free native stages with exactly one Spark partition.

Changes:

• Extend Spark 4.1 shims and offload rules to detect OneRowRelationExec and route it through a backend-specific transformer hook.
• Introduce Substrait virtual-table construction on the JVM side plus Velox conversion/validation to ValuesNode (including trivial-leaf split bookkeeping).
• Add “supportsNoInputExecution” capability and propagate it through safe operators (unary, union, streamed-side broadcast joins), with broad JVM + native test coverage and documentation updates.

File summaries

File	Description
shims/spark41/src/test/scala/org/apache/gluten/sql/shims/spark41/Spark41ShimsOneRowRelationSuite.scala	Adds Spark 4.1 shim tests for OneRowRelationExec recognition and false-positive rejection.
shims/spark41/src/main/scala/org/apache/gluten/sql/shims/spark41/Spark41Shims.scala	Implements isOneRowRelationExec for Spark 4.1 using the Spark 4.1 class type.
shims/common/src/main/scala/org/apache/gluten/sql/shims/SparkShims.scala	Adds cross-version shim hook isOneRowRelationExec (default false).
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenSQLQuerySuite.scala	Adjusts Spark 4.1 explain-plan expectations to account for native OneRowRelation in Velox.
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Excludes a Spark test whose plan checks conflict with Gluten’s native replacement.
gluten-substrait/src/test/scala/org/apache/gluten/substrait/rel/VirtualTableRelNodeSuite.scala	Adds unit tests for virtual-table rel creation/validation and protobuf round-tripping.
gluten-substrait/src/test/scala/org/apache/gluten/substrait/rel/VirtualTableProtoSuite.scala	Updates rationale/comments around virtual-table descriptor pinning.
gluten-substrait/src/test/scala/org/apache/gluten/execution/WholeStageNoInputSuite.scala	Adds unit tests for synthetic single-partition behavior with no non-broadcast inputs.
gluten-substrait/src/test/scala/org/apache/gluten/backendsapi/SparkPlanExecApiSuite.scala	Verifies unsupported backends preserve original Spark plans for this hook.
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/offload/OffloadSingleNodeRules.scala	Adds offload rule path to transform OneRowRelationExec via backend API hook.
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala	Introduces supportsNoInputExecution and uses it in partition discovery for whole-stage execution.
gluten-substrait/src/main/scala/org/apache/gluten/execution/UnionExecTransformer.scala	Propagates no-input capability across union when all children are eligible.
gluten-substrait/src/main/scala/org/apache/gluten/execution/BroadcastNestedLoopJoinExecTransformer.scala	Propagates no-input capability from the streamed side for nested-loop broadcast join.
gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala	Adds backend hook genOneRowRelationExecTransformer (default no-op).
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/VirtualTableRelNode.java	Implements JVM-side Substrait virtual-table ReadRel construction.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/RelBuilder.java	Adds makeVirtualTableReadRel helper and registers rel-to-operator mapping.
docs/velox-backend-support-progress.md	Documents ReadRel.VirtualTable and Spark 4.1 OneRowRelationExec support status/constraints.
cpp/velox/tests/Substrait2VeloxValuesNodeConversionTest.cc	Adds native tests for zero-column values, split isolation, and validation behaviors.
cpp/velox/tests/data/substrait_virtualTable_emptySchema.json	Adds test JSON plan for empty-schema virtual table inputs.
cpp/velox/substrait/SubstraitToVeloxPlan.cc	Converts Substrait virtual tables to Velox ValuesNode with validation and trivial-leaf split info.
backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala	Expands integration coverage for OneRowRelation across AQE/codegen/union/joins and mixed scans.
backends-velox/src/main/scala/org/apache/gluten/execution/OneRowRelationExecTransformer.scala	Adds Velox leaf transformer emitting a zero-column, one-row Substrait virtual table.
backends-velox/src/main/scala/org/apache/gluten/execution/HashJoinExecTransformer.scala	Propagates no-input capability from streamed side for broadcast hash join.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala	Implements Velox-specific OneRowRelation transformer generation with tag preservation.

Review details

• Files reviewed: 24/24 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.