Spark 4.2 adds a "SQL Last Attempt Metric" (SLAM) feature and test suites that assert on the metric value recorded for a query's executed plan.
When Comet is enabled, Comet replaces the Spark operators with native equivalents and the SLAM metric value is not propagated through the plan, so lastAttemptValueForDataset / lastAttemptValueForHighestRDDId return None.
Affected Spark SQL test suites (currently marked IgnoreComet / IgnoreCometSuite in dev/diffs/4.2.0.diff):
SQLLastAttemptMetricPlanShapesSuite (whole suite)MetricsFailureInjectionSuite (whole suite)SQLLastAttemptMetricIntegrationSuite ("dataset updates")
Spark 4.2 adds a "SQL Last Attempt Metric" (SLAM) feature and test suites that assert on the metric value recorded for a query's executed plan.
When Comet is enabled, Comet replaces the Spark operators with native equivalents and the SLAM metric value is not propagated through the plan, so
lastAttemptValueForDataset/lastAttemptValueForHighestRDDIdreturnNone.Affected Spark SQL test suites (currently marked
IgnoreComet/IgnoreCometSuiteindev/diffs/4.2.0.diff):SQLLastAttemptMetricPlanShapesSuite(whole suite)MetricsFailureInjectionSuite(whole suite)SQLLastAttemptMetricIntegrationSuite("dataset updates")