test: cover unsorted contiguous groups in one partition by xavlee · Pull Request #24737 · apache/datafusion

xavlee · 2026-08-27T18:54:00Z

Which issue does this PR relate to?

Part of Supporting analytics over large amounts of pre‑partitioned data #24438.

Rationale for this change

A source can concatenate several sorted logical runs into one DataFusion output partition. The resulting stream may be globally unsorted while every distinct (key, time_bin) tuple still occupies one contiguous range.

This PR records how aggregate planning handles that layout before grouped input properties are available. It provides the behavioral baseline for the remaining PRs in the stack.

What changes are included in this PR?

Add a single-partition TestMemoryExec fixture containing two sorted logical runs whose (key, time_bin) order resets at the record-batch boundary.
Aggregate by the complete (key, time_bin) tuple and verify the result.
Assert that planning selects InputOrderMode::Linear, EmissionType::Final, and SingleHashAggregateStream.

Stack

Are these changes tested?

The new aggregate test executes the single-partition input and snapshot-checks all four grouped sums.

Are there any user-facing changes?

No.

Review this layer

View only this PR layer

codecov-commenter · 2026-08-27T19:53:00Z

Codecov Report

❌ Patch coverage is 81.03448% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.44%. Comparing base (ebdb657) to head (36969e7).

Files with missing lines	Patch %	Lines
datafusion/physical-plan/src/aggregates/mod.rs	81.03%	3 Missing and 8 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24737      +/-   ##
==========================================
- Coverage   82.45%   82.44%   -0.01%     
==========================================
  Files        1140     1140              
  Lines      436434   436492      +58     
  Branches   436434   436492      +58     
==========================================
+ Hits       359840   359879      +39     
- Misses      54840    54848       +8     
- Partials    21754    21765      +11

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gene-bordegaray

approved with some suggstions for displaying expected behavior

gene-bordegaray

this guy looking good now, thank you @xavlee 💃

NGA-TRAN

LGTM

gene-bordegaray · 2026-09-22T15:21:24Z

rebumping this, just went over again and looks good.

xudong963 · 2026-09-28T02:27:24Z

Thanks

## Which issue does this PR relate to? - Part of apache#24438. ## Rationale for this change A source can concatenate several sorted logical runs into one DataFusion output partition. The resulting stream may be globally unsorted while every distinct `(key, time_bin)` tuple still occupies one contiguous range. This PR records how aggregate planning handles that layout before grouped input properties are available. It provides the behavioral baseline for the remaining PRs in the stack. ## What changes are included in this PR? - Add a single-partition `TestMemoryExec` fixture containing two sorted logical runs whose `(key, time_bin)` order resets at the record-batch boundary. - Aggregate by the complete `(key, time_bin)` tuple and verify the result. - Assert that planning selects `InputOrderMode::Linear`, `EmissionType::Final`, and `SingleHashAggregateStream`. ## Stack 1. [apache#24737 — test: cover unsorted contiguous groups in one partition](apache#24737) ← **this PR** 2. [apache#24697 — refactor: separate aggregate group completion from input ordering](apache#24697) 3. [apache#24698 — feat: add grouped equivalence properties](apache#24698) 4. [apache#24497 — feat: stream aggregates over grouped input](apache#24497) ## Are these changes tested? The new aggregate test executes the single-partition input and snapshot-checks all four grouped sums. ## Are there any user-facing changes? No. ## Review this layer [View only this PR layer](apache@36969e7)

…ache#24697) ## Which issue does this PR relate to? - Part of apache#24438. - Builds on the baseline test in apache#24737 (merged). ## Rationale for this change Aggregate execution currently uses `InputOrderMode` to select execution paths and recognize completed groups. A group can be completed incrementally when its rows form one contiguous run: once the run ends, that group cannot appear again in the same input partition. Sorted input supplies this guarantee, and unsorted input can supply it too: ```text AAABBBCCCC -> sorted; groups can be completed incrementally CCCAAABBB -> unsorted; groups can still be completed incrementally ``` This PR describes the input's _**clustering**_ guarantee with `GroupClusteringMode` and uses clustering terminology throughout the aggregate execution machinery. Aggregate construction derives the mode directly from the existing ordering analysis. The follow-up in apache#24497 will also derive it from grouped equivalence properties. ## What changes are included in this PR? - Replace the aggregate's `InputOrderMode` field with `GroupClusteringMode::{None, Partial, Full}`: - `None`: Groups have no apparent clustering guarantee. - `Partial(indices)`: rows are contiguous on a subset of grouping expressions; a change in that tuple completes all groups in the preceding run. - `Full`: rows are contiguous on the complete grouping tuple; a change in that tuple completes the preceding group. - Rename the stateful `GroupOrdering` tracker and its `Partial` and `Full` implementations to `GroupClustering`, `GroupClusteringPartial`, and `GroupClusteringFull`. - Rename the ordered aggregate streams, shared table, and adjacent-key group-values implementation to use clustered terminology, - Preserve actual sort expressions and options through equivalence properties and input-ordering requirements. - Display `group_clustering_mode` in aggregate plans and document the public API migration in the DataFusion 56 upgrade guide. ## Stack 1. [apache#24737 — test: cover unsorted contiguous groups in one partition](apache#24737) (merged) 2. [apache#24697 — refactor: use group clustering terminology in aggregate execution](apache#24697) ← **this PR** 3. [apache#24698 — feat: add Grouped equivalence properties](apache#24698) 4. [apache#24497 — feat: stream aggregates over grouped input](apache#24497) ## Are there any user-facing changes? Yes. This PR changes public Rust APIs: - `AggregateExec::input_order_mode()` is replaced by `AggregateExec::group_clustering_mode()`, returning the public `GroupClusteringMode` enum. - `AggregateExec::compute_properties` accepts `&GroupClusteringMode` in place of `&InputOrderMode`. - The public `GroupOrdering`, `GroupOrderingPartial`, and `GroupOrderingFull` types in `aggregates::order` are renamed to `GroupClustering`, `GroupClusteringPartial`, and `GroupClusteringFull`. - `GroupClustering::try_new` takes `&GroupClusteringMode`; `new_group_values` takes `&GroupClustering`. Aggregate plan output uses `group_clustering_mode=Full` and `group_clustering_mode=Partial(indices)` in place of `ordering_mode=Sorted` and `ordering_mode=PartiallySorted(indices)`. SQL results and aggregate execution-path selection retain their existing behavior. ## Review this layer [View this PR's diff](https://github.com/apache/datafusion/pull/24697/files)

## Which issue does this PR relate to? - Part of apache#24438. - Builds on apache#24737 and apache#24697, both merged. ## Rationale for this change Aggregation can recognize completed groups when all rows with the same grouping-key tuple occur in one contiguous run within a partition. Sorted input provides this guarantee, and the runs can also appear in an arbitrary order: ```text AAABBBCCCC -> ordered and grouped CCCAAABBB -> grouped; each group can be completed when its run ends AAABBBAACC -> A appears in separate runs ``` This PR represents that guarantee in `EquivalenceProperties`, allowing execution-plan producers to declare grouping tuples and consumers to check whether an input satisfies a grouping requirement. ## What changes are included in this PR? ### Semantics and propagation A grouping assertion describes a complete expression tuple within every output partition. `[a, b]` and `[b, a]` identify the same groups. The guarantee applies to the complete pair; a guarantee for a subset such as `[a]` requires its own proof. Grouping assertions are correctness guarantees supplied by producers. DataFusion trusts the asserted row layout. Producers can declare derived tuples such as `[bhandle, date_bin(timestamp)]` directly. - Store grouping tuples alongside ordering, equality, and constant information in `EquivalenceProperties`. - Normalize tuples using expression equivalences and constants, removing duplicate expressions. - Check grouping requirements against declared tuples and grouping implied by lexicographical ordering prefixes. - Project a tuple when every expression can be represented in the output, including explicitly declared derived expressions. - Remap grouping expressions during schema replacement. - Preserve a declared grouping tuple through `UnionExec` when every input satisfies it, explicitly or through ordering. - Clear explicit grouping assertions at aggregation, sorting, repartitioning, and partition-merging boundaries where execution can invalidate them. ### Public API additions - `GroupingEquivalenceClass` represents complete grouping tuples and supports construction, insertion, lookup, clearing, schema rewriting, iteration, and display. - `EquivalenceProperties::{geq_class, normalized_geq_class, add_grouping, add_groupings, clear_groupings, grouping_satisfy}` expose grouping metadata and its analysis. - `TestMemoryExec::try_with_grouping_information` attaches grouping assertions in physical-plan tests and examples. Execution-plan producers attach these assertions through the `EquivalenceProperties` held by `PlanProperties`. The follow-up apache#24497 consumes them to derive `GroupClusteringMode::Full` for eligible aggregates. ## Stack 1. [apache#24737 — test: cover unsorted contiguous groups in one partition](apache#24737) (merged) 2. [apache#24697 — refactor: use group clustering terminology in aggregate execution](apache#24697) (merged) 3. [apache#24698 — feat: add grouping equivalence properties](apache#24698) ← **this PR** 4. [apache#24497 — feat: stream aggregates over grouped input](apache#24497) ## What is the testing strategy for this PR? Tests cover complete-tuple semantics, equivalent permutations, normalization through equivalences and constants, ordering-derived grouping, complete and incomplete projection mappings, explicitly declared derived expressions, operator boundaries, and UNION propagation with selective tuple retention, mixed grouping/ordering proofs, schema rewrites, and input permutations. `aggregate_clears_explicit_grouping` covers partial and single aggregation with reordered and aliased grouping keys. Both cases fail without the aggregate boundary clear and pass with it. Local validation passed the extended workspace suite, including 2,997 physical-plan tests and 546 SQL logic files, and all-target/all-feature Clippy with warnings denied. ## Are there any user-facing changes? Yes. The additive Rust APIs above let plan producers describe contiguous grouping tuples and let consumers evaluate grouping requirements. The streaming aggregate optimization follows in apache#24497. ## Review this layer [View this PR's diff](https://github.com/apache/datafusion/pull/24698/files)

Backport the Apache DataFusion apache#24438 stack onto DataDog branch-55: - apache#24737: unsorted contiguous-group regression coverage - apache#24697: group-clustering terminology and aggregate APIs - apache#24698: grouping equivalence properties and propagation - apache#24497: incremental aggregation over explicitly grouped input - apache#26852: invalidate grouping assertions when clearing orderings Adapt the changes to DF55's existing aggregate streams. Single-stage clustered aggregation continues to use GroupedHashAggregateStream; partial and final aggregation retain the existing specialized streams. Do not import the unrelated DF56 aggregate-stream migration. Include the constant-key, grouping-set, and PartialReduce guards. Extend blocked-input tests across single/final and migrated/legacy execution, and retain sort-option and metadata-invalidation coverage. Update DF55 plan snapshots, the wrapped memory-limit error assertion, and the API upgrade guide. Validation: 503 SQL test files passed, 3347 focused expression/plan unit tests passed, and all-targets/all-features clippy passed. Extended tests exposed a dictionary sort-fuzzer failure also reproduced on the unchanged branch-55 base (root seed 1793724857; dataset 15872832093508421016; query 12465732596605393044; config 11717754459111916353). Upstream-Commit: 59d74ca Upstream-Commit: d93699c Upstream-Commit: 320a7d2 Upstream-Commit: 64a5955 Upstream-Commit: 3c3898e Upstream-Commit: 86f93b6 Co-authored-by: xavlee <52435309+xavlee@users.noreply.github.com>

Backport apache#24737 Add the baseline regression test before enabling grouped-input streaming. Upstream-Commit: 59d74ca

github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 27, 2026

This was referenced Aug 27, 2026

feat: stream aggregates over grouped input #24497

Open

refactor: use group clustering terminology in aggregate execution #24697

Merged

feat: add grouping equivalence properties #24698

Merged

xavlee force-pushed the test/issue-24438-group-contiguous-single-partition branch from 61c082c to 060dd38 Compare August 27, 2026 19:16

xavlee force-pushed the test/issue-24438-group-contiguous-single-partition branch from 060dd38 to 3b6d7ab Compare August 27, 2026 20:54

NGA-TRAN mentioned this pull request Aug 31, 2026

Supporting analytics over large amounts of pre‑partitioned data #24438

Open

gene-bordegaray approved these changes Aug 31, 2026

View reviewed changes

Comment thread datafusion/physical-plan/src/aggregates/mod.rs Resolved

Comment thread datafusion/physical-plan/src/aggregates/mod.rs Resolved

xavlee force-pushed the test/issue-24438-group-contiguous-single-partition branch from 3b6d7ab to 5dfc10b Compare August 31, 2026 17:50

gene-bordegaray approved these changes Aug 31, 2026

View reviewed changes

xudong963 approved these changes Sep 3, 2026

View reviewed changes

NGA-TRAN approved these changes Sep 3, 2026

View reviewed changes

xavlee force-pushed the test/issue-24438-group-contiguous-single-partition branch 3 times, most recently from 7100fae to fef01ed Compare September 7, 2026 20:23

xavlee force-pushed the test/issue-24438-group-contiguous-single-partition branch from fef01ed to 0e9cafe Compare September 16, 2026 14:24

xavlee force-pushed the test/issue-24438-group-contiguous-single-partition branch from 0e9cafe to 7ba0a9b Compare September 22, 2026 17:44

test: cover unsorted contiguous groups in one partition

36969e7

xavlee force-pushed the test/issue-24438-group-contiguous-single-partition branch from 7ba0a9b to 36969e7 Compare September 23, 2026 02:04

xavlee marked this pull request as ready for review September 23, 2026 19:30

xudong963 added this pull request to the merge queue Sep 28, 2026

Merged via the queue into apache:main with commit 59d74ca Sep 28, 2026
42 checks passed

nuno-faria added the v56.0.0 label Oct 3, 2026

xavlee mentioned this pull request Oct 28, 2026

fix: clear groupings when clearing orderings #26852

Open

Navigation Menu

Repository navigation

test: cover unsorted contiguous groups in one partition - #24737

test: cover unsorted contiguous groups in one partition#24737
xudong963 merged 1 commit into
apache:mainfrom
xavlee:test/issue-24438-group-contiguous-single-partition

xavlee commented Aug 27, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 27, 2026 •

edited

Loading

Uh oh!

gene-bordegaray left a comment

Uh oh!

Uh oh!

Uh oh!

gene-bordegaray left a comment •

edited

Loading

Uh oh!

NGA-TRAN left a comment

Uh oh!

gene-bordegaray commented Sep 22, 2026

Uh oh!

xudong963 commented Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

xavlee commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR relate to?

Rationale for this change

What changes are included in this PR?

Stack

Are these changes tested?

Are there any user-facing changes?

Review this layer

Uh oh!

codecov-commenter commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gene-bordegaray left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NGA-TRAN left a comment

Choose a reason for hiding this comment

Uh oh!

gene-bordegaray commented Sep 22, 2026

Uh oh!

xudong963 commented Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

xavlee commented Aug 27, 2026 •

edited

Loading

codecov-commenter commented Aug 27, 2026 •

edited

Loading

gene-bordegaray left a comment •

edited

Loading