[GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation by felipepessoto · Pull Request #12967 · apache/gluten

felipepessoto · 2026-09-04T01:37:07Z

What changes are proposed in this pull request?

Enable native TIMESTAMP_NTZ aggregation in the Velox backend:

Allow aggregate and shuffle operators, plus the projection expressions used by Delta statistics collection, through the TIMESTAMP_NTZ fallback validator.
Accept TimestampNTZType aggregate buffers.
Preserve TIMESTAMP_NTZ as a distinct tsntz function-signature type and as Substrait PrecisionTimestamp.
Add native min/max and Delta statistics regression coverage, including nested TIMESTAMP_NTZ columns.
Remove 42 Delta data-skipping tests fixed by the native statistics aggregation support from the known-failure baseline.

This prevents Delta's local statistics SortAggregateExec from falling back to a vanilla Spark plan and addresses the TIMESTAMP_NTZ failure described in #12538. This change is independent of #12966.

How was this patch tested?

cpp/build/velox/tests/velox_plan_conversion_test --gtest_filter='VeloxSubstraitSignatureTest.*:VeloxToSubstraitTypeTest.*:VeloxSubstraitRoundTripTest.minMaxTimestampUtc'
./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta -DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/" -DwildcardSuites=org.apache.spark.sql.GlutenTimestampNtzAggregateSuite
./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta -DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/" -DwildcardSuites=org.apache.spark.sql.delta.GlutenDeltaStatsSuite
./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-3.5 -Pscala-2.12 -Pjava-17 -Pdelta -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" -DwildcardSuites=org.apache.spark.sql.delta.GlutenDeltaStatsSuite
./build/mvn test -Pspark-ut -Pbackends-velox -Pspark-3.5 -Pscala-2.12 -Pjava-17 -Pdelta -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" -DwildcardSuites=org.apache.gluten.functions.DateFunctionsValidateSuite

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI 1.0.82

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-04T01:37:36Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-09-04T01:37:37Z

🔄 Delta Spark UT started by @felipepessoto (~2.5 h). View run

github-actions · 2026-09-04T01:39:45Z

Run Gluten Clickhouse CI on x86

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-04T01:49:16Z

Run Gluten Clickhouse CI on x86

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-04T02:49:25Z

Run Gluten Clickhouse CI on x86

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-04T02:54:45Z

Run Gluten Clickhouse CI on x86

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-04T04:58:16Z

Run Gluten Clickhouse CI on x86

Copilot

🟡 Changes recommended

A newly added Velox round-trip test uses millisecond-precision timestamps despite declaring microsecond precision, weakening coverage for the intended NTZ precision behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Enables native handling of TIMESTAMP_NTZ (Spark) / TIMESTAMP_UTC (Velox) in aggregation and related Delta statistics collection paths for the Velox backend, reducing Spark fallbacks and unblocking Delta stats plans previously rejected due to NTZ types.

Changes:

Extend the TimestampNTZ fallback validator to allow aggregates, shuffles, and direct NTZ projections needed by Delta stats plans.
Accept TimestampNTZType in aggregation buffer type checks and preserve a distinct native signature token (tsntz) plus Substrait PrecisionTimestamp encoding.
Add native regression coverage (Spark UT + Velox C++ tests) and remove now-fixed Delta data-skipping cases from the known-failure baseline.

File summaries

File	Description
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala	Adds Spark 4.1 regression coverage for NTZ min/max aggregation and a projection fallback case.
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala	Enables the new Spark 4.1 UT suite in Velox test settings.
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala	Broadens NTZ fallback validation to permit additional plan nodes/expressions used by native stats aggregation.
gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala	Maps Spark `TimestampNTZType` to Substrait timestamp NTZ type node and uses `tsntz` in signature naming.
gluten-substrait/src/main/scala/org/apache/gluten/execution/HashAggregateExecBaseTransformer.scala	Allows `TimestampNTZType` in supported aggregation buffer type checks.
cpp/velox/tests/VeloxToSubstraitTypeTest.cc	Adds a type-conversion test for `TIMESTAMP_UTC` -> Substrait `PrecisionTimestamp`.
cpp/velox/tests/VeloxSubstraitSignatureTest.cc	Adds `tsntz` signature coverage for `TIMESTAMP_UTC` mapping in both directions.
cpp/velox/tests/VeloxSubstraitRoundTripTest.cc	Adds a min/max aggregation round-trip test over `TIMESTAMP_UTC`.
cpp/velox/substrait/VeloxToSubstraitType.cc	Encodes `TIMESTAMP_UTC` as Substrait `precision_timestamp(6)`.
cpp/velox/substrait/VeloxSubstraitSignature.cc	Maps `TIMESTAMP_UTC` to/from the `tsntz` signature token.
backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala	Adds Delta 4.0 stats regression coverage including nested NTZ columns.
backends-velox/src-delta33/test/scala/org/apache/spark/sql/delta/GlutenDeltaStatsSuite.scala	Adds Delta 3.3 stats regression coverage including nested NTZ columns.
.github/workflows/util/delta-spark-ut/known-failures.txt	Removes Delta data-skipping known failures resolved by native NTZ stats aggregation support.

Review details

Files reviewed: 13/13 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

felipepessoto · 2026-09-04T09:44:37Z

@Mariamalmesfer @rui-mo I think you two have worked on other timestamp_ntz PRs. Could you take a look, please?

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

github-actions · 2026-09-04T18:38:07Z

Run Gluten Clickhouse CI on x86

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-04T19:20:51Z

Run Gluten Clickhouse CI on x86

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:1

The previous implementation used dataType.typeName == "timestamp_ntz", which is resilient across Spark versions/shims. Switching to a direct TimestampNTZType reference introduces a compile-time dependency that can break builds for Spark variants where TimestampNTZType is absent or shaded differently. If this module is cross-built across multiple Spark versions, consider keeping a version-tolerant check (e.g., match TimestampNTZType when available and fall back to typeName), ideally via a shim utility.

/*

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-04T20:13:53Z

Run Gluten Clickhouse CI on x86

Copilot

🟢 Approval recommended

The changes are cohesive and well-covered by targeted Spark + native regression tests, and the updated validator/signature/type mappings are consistent across JVM and C++ paths.

Review details

Files reviewed: 13/13 changed files
Comments generated: 0 new
Review effort level: Lite

github-actions · 2026-09-04T22:38:36Z

Run Gluten Clickhouse CI on x86

rui-mo

Thanks @felipepessoto. Could you please clarify whether this implementation is based on the aggregate implementation for the Timestamp type in Velox, and whether it can be fully reused for the TimestampNTZ type?

rui-mo · 2026-09-07T14:23:52Z

+              "1969-12-31T23:59:59.999",
+              "2024-01-01T00:00:00.123",
+              "2024-01-01T00:00:00.123"),
+            stats)


Could you please assert that the plan contains HashAggregateExecTransformer to ensure aggregate has been offloaded into native?

rui-mo · 2026-09-07T14:24:47Z

    return "date";
  }
+  if (type->equivalent(*TIMESTAMP_UTC())) {
+    return "tsntz";


nit: use ts_ntz to be aligned with Spark.

[GLUTEN-12538][VL] Support TIMESTAMP_NTZ aggregation

fae12eb

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions Bot added CORE works for Gluten Core VELOX labels Sep 4, 2026

felipepessoto closed this Sep 4, 2026

felipepessoto reopened this Sep 4, 2026

[MINOR] Document TIMESTAMP_NTZ signature token

53c59c5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Fix NTZ aggregate validation coverage

ecc6510

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[GLUTEN-12538][VL] Update Delta test baseline

b7ca17e

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions Bot added the INFRA label Sep 4, 2026

[GLUTEN-12538][VL] Restore unrelated Delta failures

243f818

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

felipepessoto marked this pull request as ready for review September 4, 2026 09:33

Copilot AI lite review requested due to automatic review settings September 4, 2026 09:33

Copilot started reviewing on behalf of felipepessoto September 4, 2026 09:39 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread cpp/velox/tests/VeloxSubstraitRoundTripTest.cc Outdated

[GLUTEN-12538][VL] Test microsecond NTZ round trip

e6e42bd

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings September 4, 2026 17:57

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Copilot started reviewing on behalf of felipepessoto September 4, 2026 18:38 View session

felipepessoto mentioned this pull request Sep 4, 2026

[GLUTEN-12538][VL] Fall back to row-based Delta stats when the stats plan cannot be offloaded #12539

Closed

[GLUTEN-12538][VL] Address follow-up review comments

bcfffcd

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings September 4, 2026 19:06

Copilot started reviewing on behalf of felipepessoto September 4, 2026 19:21 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread cpp/velox/substrait/VeloxSubstraitSignature.cc

Comment thread gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenTimestampNtzAggregateSuite.scala

[GLUTEN-12538][VL] Localize NTZ fallback assertion

6f80ab9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings September 4, 2026 20:13

Copilot started reviewing on behalf of felipepessoto September 4, 2026 20:13 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

felipepessoto closed this Sep 4, 2026

felipepessoto reopened this Sep 4, 2026

rui-mo reviewed Sep 7, 2026

View reviewed changes

Uh oh!

Conversation

felipepessoto commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

felipepessoto commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

rui-mo left a comment

Choose a reason for hiding this comment

Uh oh!

rui-mo Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

rui-mo Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

felipepessoto commented Sep 4, 2026 •

edited

Loading