Spark 4.2 adds a segment-tree window-aggregation optimization to WindowExec, along with metrics such as "number of segment-tree frames prepared" and "number of segment-tree fallback frames prepared", plus an eligibility allowlist.
Comet replaces WindowExec with CometWindowExec, which has no segment-tree implementation and does not expose those metrics. The new test suites read the metric from a Spark Window node in the executed plan and fail with "No Window node in plan" when Comet is enabled.
Affected Spark SQL test suites (currently marked IgnoreCometSuite in dev/diffs/4.2.0.diff):
SegmentTreeWindowMetricsSuiteWindowSegmentTreeAllowlistSuite
Spark 4.2 adds a segment-tree window-aggregation optimization to
WindowExec, along with metrics such as "number of segment-tree frames prepared" and "number of segment-tree fallback frames prepared", plus an eligibility allowlist.Comet replaces
WindowExecwithCometWindowExec, which has no segment-tree implementation and does not expose those metrics. The new test suites read the metric from a SparkWindownode in the executed plan and fail with "No Window node in plan" when Comet is enabled.Affected Spark SQL test suites (currently marked
IgnoreCometSuiteindev/diffs/4.2.0.diff):SegmentTreeWindowMetricsSuiteWindowSegmentTreeAllowlistSuite