From c6d18768b3744db78e8ccdf42016cdc55aae78f7 Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Thu, 6 Aug 2026 04:51:07 +0000 Subject: [PATCH 1/9] [SPARK-58611][SS] Left anti stream-stream join ### What changes were proposed in this pull request? This adds LeftAnti support to stream-stream join, which previously failed at analysis time with "LeftAnti joins with a streaming DataFrame/Dataset on the right are not supported". Unlike left semi, left anti cannot emit while joining: a semi match is positively determined, whereas "no match exists" is only decidable once the watermark guarantees no future right row can match. Left anti is therefore implemented on the eviction path, reusing the existing left outer plumbing in `StreamingSymmetricHashJoinExec`: * nothing is emitted when a left row matches; * at left-side state eviction, rows whose `matched` flag is false are emitted as bare left rows (left outer emits them joined with nulls instead); * a matched left row stays in state carrying `matched = true` so that it is suppressed at eviction time, rather than being dropped from state early the way left semi does; * a left row that fails the pre-join filter can never match, so it is emitted immediately without being added to state. Two details worth calling out for review: * `AddingProcessedRowToStateCompletionIterator` infers the persisted `matched` flag from whether the output iterator is non-empty. Left anti emits nothing on a match, so the match status is now passed explicitly via a new optional `matchedOverride` parameter. Without it every left row would be stored as unmatched and matched rows would be wrongly emitted at eviction. The parameter defaults to the previous behaviour, so the other join types are unaffected. * The joined-row iterator is drained fully rather than short-circuited on the first match, because `getJoinedRows` sets the `matched` flag on the other side's rows lazily as they are produced. Stopping early would leave some matched left rows flagged as unmatched. Requirements, mirroring left outer: a watermark on the right side plus time constraints are mandatory, and Append is the only supported output mode (Update would have to emit rows before the watermark can rule out a future match, and such a row could be invalidated by a later batch). No new state format version is needed -- the `matched` flag already persisted by v2/v3/v4 is exactly the required signal, so existing checkpoints need no migration. RightAnti remains out of scope. ### Why are the changes needed? Stream-stream join supported Inner, LeftOuter, RightOuter, FullOuter and LeftSemi. LeftSemi was added in SPARK-32862 but its complement was never done, leaving the common "find left rows with no match on the right" pattern -- impressions without clicks, orders without shipments, sessions without conversion -- without a native streaming implementation. Users work around it with NOT IN / NOT EXISTS rewrites or hand-written transformWithState logic, both more expensive and easy to get subtly wrong. Note stream-static LEFT ANTI already worked when only the left side was streaming; only a streaming right side was rejected, so the gap was specifically stream-stream. ### Does this PR introduce _any_ user-facing change? Yes. `LEFT ANTI` stream-stream joins are now supported in Append output mode, given a watermark on the right side and time constraints. Queries which previously failed at analysis time now run. No existing behaviour changes. Left anti buffers every left row until eviction, whereas left semi drops matched left rows from state eagerly. This is inherent -- a matched row must be retained so that it can be suppressed at eviction rather than emitted -- so state size for left anti is comparable to left outer, not to left semi. The join support matrix in the Structured Streaming guide gains Left Anti rows for stream-static, static-stream and stream-stream, plus an "Anti Joins with Watermarking" section. ### How was this patch tested? New `StreamingLeftAntiJoinSuite` with virtual-column-family and non-VCF variants, covering windowed anti join across restarts, an unmatched row only being emitted once the watermark passes it, a row matched in a later batch never being emitted, pre-join-filter exclusion on both sides, and Update output mode being rejected. `UnsupportedOperationsSuite` gains LeftAnti coverage for the watermark conditions and for Update/Complete mode rejection. Verified locally: `UnsupportedOperationsSuite` 226/226 pass; the new left anti suite plus the existing left semi suite 32/32 pass on both RocksDB and HDFS-backed state store providers. --- .../apis-on-dataframes-and-datasets.md | 43 ++++- .../UnsupportedOperationChecker.scala | 19 +- .../analysis/UnsupportedOperationsSuite.scala | 16 +- .../join/StreamingSymmetricHashJoinExec.scala | 109 ++++++++++- .../sql/streaming/StreamingJoinSuite.scala | 177 +++++++++++++++++- 5 files changed, 334 insertions(+), 30 deletions(-) diff --git a/docs/streaming/apis-on-dataframes-and-datasets.md b/docs/streaming/apis-on-dataframes-and-datasets.md index 86585caead51f..a133fc112ac9b 100644 --- a/docs/streaming/apis-on-dataframes-and-datasets.md +++ b/docs/streaming/apis-on-dataframes-and-datasets.md @@ -1322,6 +1322,22 @@ side in future. Semi joins have the same guarantees as [inner joins](#semantic-guarantees-of-stream-stream-inner-joins-with-watermarking) regarding watermark delays and whether data will be dropped or not. +##### Anti Joins with Watermarking +An anti join returns values from the left side of the relation that has no match with the right. +It is also referred to as a left anti join. As with semi joins, watermark + event-time constraints +must be specified for an anti join: since a row is emitted precisely because it has *no* match, the +engine has to wait until the watermark guarantees that no matching row can arrive on the right side +in future before it can emit the row. + +Note that anti join is only supported in Append output mode. Update mode would have to emit rows +early, before the watermark can rule out a future match, and such a row could be invalidated by a +later batch. + +###### Semantic Guarantees of Stream-stream Anti Joins with Watermarking +Anti joins have the same guarantees regarding watermark delays and whether data will be dropped as +[outer joins](#outer-joins-with-watermarking), because unmatched rows are likewise only emitted once +the watermark has passed them. + ##### Support matrix for joins in streaming queries @@ -1343,8 +1359,8 @@ regarding watermark delays and whether data will be dropped or not. - - + + @@ -1365,8 +1381,12 @@ regarding watermark delays and whether data will be dropped or not. - - + + + + + + @@ -1387,8 +1407,12 @@ regarding watermark delays and whether data will be dropped or not. - - + + + + + + + + + + diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala index eddcee169e377..363cb10b75f10 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala @@ -491,7 +491,10 @@ object UnsupportedOperationChecker extends Logging { joinType match { // The behavior for unmatched rows in outer joins with update mode // hasn't been defined yet. - case LeftOuter | RightOuter | FullOuter => + // LeftAnti is included here because its unmatched rows are only emitted once the + // watermark guarantees no future match, so early-firing in Update mode would + // produce rows which a later batch could invalidate. + case LeftOuter | RightOuter | FullOuter | LeftAnti => if (outputMode != InternalOutputModes.Append) { throwError(s"$joinType join between two streaming DataFrames/Datasets" + s" is not supported in ${outputMode} output mode, only in Append output mode") @@ -522,10 +525,16 @@ object UnsupportedOperationChecker extends Logging { checkForStreamStreamJoinWatermark(j) } + // We support streaming left anti joins with stream on both sides under the + // appropriate conditions. A streaming right with a static left is not supported: + // unmatched left rows are determined at watermark-based eviction of the left state, + // which a static left side does not have. case LeftAnti => - if (right.isStreaming) { - throwError(s"$LeftAnti joins with a streaming DataFrame/Dataset " + - "on the right are not supported") + if (!left.isStreaming && right.isStreaming) { + throwError(s"$LeftAnti join with a streaming DataFrame/Dataset " + + "on the right and a static DataFrame/Dataset on the left is not supported") + } else if (left.isStreaming && right.isStreaming) { + checkForStreamStreamJoinWatermark(j) } // We support streaming left outer and left semi joins with static on the right always, @@ -687,7 +696,7 @@ object UnsupportedOperationChecker extends Logging { // Check if the nullable side has a watermark, and there's a range condition which // implies a state value watermark on the first side. val hasValidWatermarkRange = join.joinType match { - case LeftOuter | LeftSemi => StreamingJoinHelper.getStateValueWatermark( + case LeftOuter | LeftSemi | LeftAnti => StreamingJoinHelper.getStateValueWatermark( join.left.outputSet, join.right.outputSet, join.condition, Some(1000000)).isDefined case RightOuter => StreamingJoinHelper.getStateValueWatermark( join.right.outputSet, join.left.outputSet, join.condition, Some(1000000)).isDefined diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala index 293523b86f998..ae114e76f0aa9 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala @@ -424,7 +424,8 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { streamBatchSupported = false, expectedMsg = "FullOuter join") - // Left outer, left semi, left anti join: *-stream not allowed + // Left outer, left semi, left anti join: batch-stream not allowed, and stream-stream join is + // allowed 'conditionally' - see the watermark checks below Seq((LeftOuter, "LeftOuter join"), (LeftSemi, "LeftSemi join"), (LeftAnti, "LeftAnti join")) .foreach { case (joinType, name) => testBinaryOperationInStreamingPlan( @@ -443,8 +444,10 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { streamStreamSupported = false, expectedMsg = "RightOuter join") - // Left outer, right outer, full outer joins: Update mode not allowed - Seq(LeftOuter, RightOuter, FullOuter).foreach { joinType => + // Left outer, right outer, full outer, left anti joins: Update mode not allowed. Left anti is + // included because its unmatched rows are only emitted at watermark-based eviction, so + // early-firing could emit a row which a later batch would invalidate. + Seq(LeftOuter, RightOuter, FullOuter, LeftAnti).foreach { joinType => assertNotSupportedInStreamingPlan( s"$joinType join with stream-stream relations and update mode", streamRelation.join(streamRelation, joinType = joinType, @@ -467,7 +470,8 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { (LeftSemi, "only in Append and Update output modes"), (LeftOuter, "only in Append output mode"), (RightOuter, "only in Append output mode"), - (FullOuter, "only in Append output mode") + (FullOuter, "only in Append output mode"), + (LeftAnti, "only in Append output mode") ).foreach { case (joinType, allowedModesMsg) => assertNotSupportedInStreamingPlan( s"$joinType join with stream-stream relations and complete mode", @@ -477,8 +481,8 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { Seq("is not supported in Complete output mode", allowedModesMsg)) } - // Left outer, right outer, full outer, left semi joins - Seq(LeftOuter, RightOuter, FullOuter, LeftSemi).foreach { joinType => + // Left outer, right outer, full outer, left semi, left anti joins + Seq(LeftOuter, RightOuter, FullOuter, LeftSemi, LeftAnti).foreach { joinType => // Stream-stream allowed with join on watermark attribute // Note that the attribute need not be watermarked on both sides. assertSupportedInStreamingPlan( diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala index 8f90a603c7efb..de0e33edee84c 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala @@ -187,7 +187,7 @@ case class StreamingSymmetricHashJoinExec( require( joinType == Inner || joinType == LeftOuter || joinType == RightOuter || joinType == FullOuter || - joinType == LeftSemi, + joinType == LeftSemi || joinType == LeftAnti, errorMessageForJoinType) outputMode.foreach { mode => @@ -238,7 +238,7 @@ case class StreamingSymmetricHashJoinExec( case LeftOuter => left.output ++ right.output.map(_.withNullability(true)) case RightOuter => left.output.map(_.withNullability(true)) ++ right.output case FullOuter => (left.output ++ right.output).map(_.withNullability(true)) - case LeftSemi => left.output + case LeftSemi | LeftAnti => left.output case _ => throwBadJoinTypeException() } WidenStatefulOpNullability.widenOutputForStatefulOp(base) @@ -251,7 +251,7 @@ case class StreamingSymmetricHashJoinExec( case LeftOuter => left.outputPartitioning case RightOuter => right.outputPartitioning case FullOuter => UnknownPartitioning(left.outputPartitioning.numPartitions) - case LeftSemi => left.outputPartitioning + case LeftSemi | LeftAnti => left.outputPartitioning case _ => throwBadJoinTypeException() } @@ -422,6 +422,12 @@ case class StreamingSymmetricHashJoinExec( // For Left Semi Join, we process the right side first so that new right input is stored // before left input probes against it. This way, new left rows that match new right rows // in the same microbatch are emitted immediately without being buffered in state. + // + // Left Anti Join generates nothing while joining, on either side: whether a left row has no + // match is only decided once the watermark guarantees no future right row can match it, so + // unmatched left rows are emitted when the left side state is evicted (see `outputIter`). Both + // sides still store their input, and matching a new right input marks the corresponding stored + // left rows as matched so that they are suppressed at eviction time. val leftOutputIter = joinerManager.leftSideJoiner.storeAndJoinWithOtherSide(joinerManager.rightSideJoiner) { (input: InternalRow, matched: InternalRow) => joinedRow.withLeft(input).withRight(matched) @@ -542,10 +548,40 @@ case class StreamingSymmetricHashJoinExec( val rightSideOutputIter = new LazilyInitializingJoinedRowIterator(rightSideInitIterFn) hashJoinOutputIter ++ leftSideOutputIter ++ rightSideOutputIter + case LeftAnti => + // Left anti join is structurally an eviction-time join, like left outer: whether a left + // row has "no match" is only known once the watermark guarantees no future right row can + // match it. So we reuse the left outer eviction path, with two differences: + // * nothing is emitted when a left row matches, and + // * an unmatched left row is emitted as the bare left row rather than joined with nulls. + // + // State format version 1 is already rejected for non-inner joins (see the constructor), + // so only the 'matched' flag path needs to be handled. + val initIterFn = { () => + val removedRowIter = joinerManager.leftSideJoiner.removeAndReturnOldState() + removedRowIter.filterNot { kv => + stateFormatVersion match { + case 2 | 3 | 4 => kv.matched + case _ => throwBadStateFormatVersionException() + } + }.map(_.value) + } + + // NOTE: we need to make sure `antiOutputIter` is evaluated "after" exhausting all of + // elements in `hashJoinOutputIter`, otherwise it may lead to out of sync according to + // the interface contract on StateStore.iterator and end up with correctness issue. + // Please refer SPARK-38684 for more details. + val antiOutputIter = new LazilyInitializingRowIterator(initIterFn) + + // `hashJoinOutputIter` still has to be consumed: draining it is what appends input rows to + // the state stores and persists the 'matched' flag for left rows which found a match. For + // left anti it only carries the rows which failed the pre-join filter (see + // `generateFilteredJoinedRow`), which are genuine anti output, so it is concatenated as-is. + hashJoinOutputIter ++ antiOutputIter case _ => throwBadJoinTypeException() } - val outputProjection = if (joinType == LeftSemi) { + val outputProjection = if (joinType == LeftSemi || joinType == LeftAnti) { UnsafeProjection.create(output, output) } else { UnsafeProjection.create(left.output ++ right.output, output) @@ -587,10 +623,14 @@ case class StreamingSymmetricHashJoinExec( // // For full outer joins, we have already removed unnecessary states from both sides, so // nothing needs to be outputted here. + // + // For left anti joins, the left side state was already consumed and removed while + // generating the unmatched ("anti") output, same as the left side of a left outer join, + // so only the right side remains to be removed greedily. numRemovedStateRows += ( joinType match { case Inner | LeftSemi => joinerManager.removeOldState() - case LeftOuter => joinerManager.rightSideJoiner.removeOldState() + case LeftOuter | LeftAnti => joinerManager.rightSideJoiner.removeOldState() case RightOuter => joinerManager.leftSideJoiner.removeOldState() case FullOuter => 0L case _ => throwBadJoinTypeException() @@ -779,6 +819,9 @@ case class StreamingSymmetricHashJoinExec( (row: InternalRow) => Iterator(generateJoinedRow(row, nullRight)) case RightSide if joinType == RightOuter || joinType == FullOuter => (row: InternalRow) => Iterator(generateJoinedRow(row, nullLeft)) + // A left row which fails the pre-join filter can never satisfy the join condition, so it + // is unmatched by definition and is emitted right away without being added to the state. + case LeftSide if joinType == LeftAnti => (row: InternalRow) => Iterator(row) case _ => (_: InternalRow) => Iterator.empty } @@ -787,12 +830,15 @@ case class StreamingSymmetricHashJoinExec( // unmatched rows) and on the left side of left semi (for matched-rows removal). // For older versions, we do not apply the optimization as it is a behavioral change, // although the optimization is valid for all versions. + // + // Left anti join needs the flag on the left side rows only, so that unmatched left rows can + // be identified at eviction time. That flag is written while processing the right side. val needToUpdateMatchedOnOtherSide = joinType match { case Inner => false case LeftOuter => joinSide == RightSide case RightOuter => joinSide == LeftSide case FullOuter => true - case LeftSemi => joinSide == RightSide + case LeftSemi | LeftAnti => joinSide == RightSide case _ => true } val skipUpdatingMatchedFlag = stateFormatVersion == 4 && !needToUpdateMatchedOnOtherSide @@ -845,21 +891,48 @@ case class StreamingSymmetricHashJoinExec( timestampRange = computeTimestampRange(thisRow), skipUpdatingMatchedFlag) } - val outputIter = generateOutputIter(thisRow, joinedRowIter) - new AddingProcessedRowToStateCompletionIterator(key, thisRow, outputIter) + if (joinType == LeftAnti) { + // Left anti join emits nothing when rows match, on either side; unmatched left rows + // are emitted later during watermark-based eviction of the left side state. The match + // status is passed explicitly, since it can no longer be inferred from the (empty) + // output iterator. + // + // The iterator must be drained fully rather than short-circuited on the first match: + // `getJoinedRows` sets the 'matched' flag on the other side's rows lazily, as they are + // produced, so stopping early would leave some matched left rows flagged as unmatched + // and they would then be wrongly emitted as anti output at eviction time. Draining is + // also what the state manager API requires of callers. + var matched = false + while (joinedRowIter.hasNext) { + joinedRowIter.next() + matched = true + } + new AddingProcessedRowToStateCompletionIterator( + key, thisRow, Iterator.empty, Some(matched)) + } else { + val outputIter = generateOutputIter(thisRow, joinedRowIter) + new AddingProcessedRowToStateCompletionIterator(key, thisRow, outputIter) + } } else { generateFilteredJoinedRow(thisRow) } } } + /** + * @param matchedOverride the match status of `thisRow`, for join types whose output iterator + * does not reflect whether a match was found (left anti emits nothing + * on a match). When empty, the match status is inferred from whether + * the output iterator is non-empty. + */ private class AddingProcessedRowToStateCompletionIterator( key: UnsafeRow, thisRow: UnsafeRow, - subIter: Iterator[InternalRow]) + subIter: Iterator[InternalRow], + matchedOverride: Option[Boolean] = None) extends CompletionIterator[InternalRow, Iterator[InternalRow]](subIter) { - private val iteratorNotEmpty: Boolean = super.hasNext + private val iteratorNotEmpty: Boolean = matchedOverride.getOrElse(super.hasNext) override def completion(): Unit = { // The criteria of whether the input has to be added into state store or not: @@ -874,6 +947,10 @@ case class StreamingSymmetricHashJoinExec( // if input is going to be evicted in this batch. Though, input should be added to the // state store if it's right outer join or full outer join, as unmatched output is // handled during state eviction. + // + // Note left anti deliberately does not get the left semi skip-on-match treatment: a matched + // left row has to stay in state (carrying matched = true) so that it is suppressed, rather + // than emitted, when the left side state is evicted. val isLeftSemiWithMatch = joinType == LeftSemi && joinSide == LeftSide && iteratorNotEmpty val shouldAddToState = if (isLeftSemiWithMatch) { false @@ -1084,6 +1161,18 @@ case class StreamingSymmetricHashJoinExec( override def next(): JoinedRow = iter.next() } + /** + * Same as [[LazilyInitializingJoinedRowIterator]], but for join types which emit rows from a + * single side (left anti) rather than joined rows. + */ + private class LazilyInitializingRowIterator( + initFn: () => Iterator[UnsafeRow]) extends Iterator[UnsafeRow] { + private lazy val iter: Iterator[UnsafeRow] = initFn() + + override def hasNext: Boolean = iter.hasNext + override def next(): UnsafeRow = iter.next() + } + // If `STATE_STORE_SKIP_NULLS_FOR_STREAM_STREAM_JOINS` is enabled, counting the number // of skipped null values as custom metric of stream join operator. override def customStatefulOperatorMetrics: Seq[StatefulOperatorCustomMetric] = diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala index c0983f338abe5..4bf1a58bd639b 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala @@ -157,7 +157,7 @@ abstract class StreamingJoinSuite val windowed2 = df2 .select($"key", window($"rightTime", "10 second"), $"rightValue") val joined = windowed1.join(windowed2, Seq("key", "window"), joinType) - val select = if (joinType == "left_semi") { + val select = if (joinType == "left_semi" || joinType == "left_anti") { joined.select($"key", $"window.end".cast("long"), $"leftValue") } else { joined.select($"key", $"window.end".cast("long"), $"leftValue", @@ -185,7 +185,7 @@ abstract class StreamingJoinSuite && $"leftValue" > 4, joinType) - val select = if (joinType == "left_semi") { + val select = if (joinType == "left_semi" || joinType == "left_anti") { joined.select(left("key"), left("window.end").cast("long"), $"leftValue") } else if (joinType == "left_outer") { joined.select(left("key"), left("window.end").cast("long"), $"leftValue", @@ -219,7 +219,7 @@ abstract class StreamingJoinSuite && $"rightValue".cast("int") > 7, joinType) - val select = if (joinType == "left_semi") { + val select = if (joinType == "left_semi" || joinType == "left_anti") { joined.select(left("key"), left("window.end").cast("long"), $"leftValue") } else if (joinType == "left_outer") { joined.select(left("key"), left("window.end").cast("long"), $"leftValue", @@ -2632,6 +2632,167 @@ class StreamingFullOuterJoinWithoutVCFSuite extends StreamingFullOuterJoinSuite override protected def testMode = Mode.WithoutVCF } +abstract class StreamingLeftAntiJoinSuite extends StreamingJoinSuite { + + test("windowed left anti join") { + withTempDir { checkpointDir => + val (leftInput, rightInput, joined) = setupWindowedJoin("left_anti") + + testStream(joined, OutputMode.Append())( + StartStream(checkpointLocation = checkpointDir.getCanonicalPath), + MultiAddData(leftInput, 1, 2, 3, 4, 5)(rightInput, 3, 4, 5, 6, 7), + // Nothing is emitted yet: left 1 and 2 are unmatched, but the watermark cannot yet rule + // out a future match for them. Left 3, 4, 5 matched and are suppressed forever. + CheckNewAnswer(), + // states + // left: 1, 2, 3, 4, 5 (all buffered; 3, 4, 5 carry matched = true) + // right: 3, 4, 5, 6, 7 + assertNumStateRows( + total = Seq(10), updated = Seq(10), + droppedByWatermark = Seq(0), removed = Some(Seq(0))), + MultiAddData(leftInput, 21)(rightInput, 22), + // Watermark = 11, so window=[0,10] is evicted from the left side: the unmatched left rows + // 1 and 2 are emitted now, while the matched 3, 4, 5 are dropped without output. + CheckNewAnswer(Row(1, 10, 2), Row(2, 10, 4)), + // states + // left: 21 + // right: 22 + // + // states evicted + // left: 1, 2, 3, 4, 5 (below watermark) + // right: 3, 4, 5, 6, 7 (below watermark) + // + // Only the 5 right side rows are reported as removed: the left side rows are evicted + // through removeAndReturnOldState while generating the anti output, which does not feed + // numRemovedStateRows. Left outer join reports removals the same way. + assertNumStateRows( + total = Seq(2), updated = Seq(2), + droppedByWatermark = Seq(0), removed = Some(Seq(5))), + StopStream, + // Restart the join query from the same checkpoint + StartStream(checkpointLocation = checkpointDir.getCanonicalPath), + AddData(leftInput, 22), + // Left 22 matches right 22, so it is buffered with matched = true and never emitted. + CheckNewAnswer(), + // states + // left: 21, 22 + // right: 22 + assertNumStateRows( + total = Seq(3), updated = Seq(1), + droppedByWatermark = Seq(0), removed = Some(Seq(0))), + StopStream, + // Restart the query from the same checkpoint + StartStream(checkpointLocation = checkpointDir.getCanonicalPath), + AddData(leftInput, 1), + // Row not added as 1 < state key watermark = 12. + CheckNewAnswer(), + // states + // left: 21, 22 + // right: 22 + assertNumStateRows( + total = Seq(3), updated = Seq(0), + droppedByWatermark = Seq(1), removed = Some(Seq(0))) + ) + } + } + + test("left anti join emits an unmatched left row only once the watermark passes it") { + val (leftInput, rightInput, joined) = setupWindowedJoin("left_anti") + + testStream(joined, OutputMode.Append())( + AddData(leftInput, 3), + // Unmatched so far, but not yet provably unmatched. + CheckNewAnswer(), + // A matching right row arrives in a later batch, so left 3 must never be emitted. + AddData(rightInput, 3), + CheckNewAnswer(), + // Advance the watermark past window=[0,10]. + MultiAddData(leftInput, 21)(rightInput, 21), + CheckNewAnswer(), + // Left 21 matched right 21, and left 3 was matched before eviction, so still nothing. + MultiAddData(leftInput, 31)(rightInput, 31), + CheckNewAnswer() + ) + } + + test("left anti early state exclusion on left") { + val (leftInput, rightInput, joined) = setupWindowedJoinWithLeftCondition("left_anti") + + testStream(joined, OutputMode.Append())( + MultiAddData(leftInput, 1, 2, 3)(rightInput, 3, 4, 5), + // The left rows with leftValue <= 4 (i.e. left 1 and 2) fail the pre-join filter, so they can + // never match and are emitted immediately without being added to the state. + CheckNewAnswer(Row(1, 10, 2), Row(2, 10, 4)), + // states + // left: 3 (matched right 3 in the same batch, buffered with matched = true) + // right: 3, 4, 5 + assertNumStateRows( + total = Seq(4), updated = Seq(4), + droppedByWatermark = Seq(0), removed = Some(Seq(0))), + // Left 3 matched, so advancing the watermark must not produce an anti row for it. + MultiAddData(leftInput, 20)(rightInput, 21), + CheckNewAnswer(), + // states + // left: 20 + // right: 21 + // + // states evicted + // left: 3 (below watermark, matched so no output) + // right: 3, 4, 5 (below watermark) + // + // Only the 3 right side rows are counted as removed - see the note in + // "windowed left anti join" about left side eviction not feeding numRemovedStateRows. + assertNumStateRows( + total = Seq(2), updated = Seq(2), + droppedByWatermark = Seq(0), removed = Some(Seq(3))) + ) + } + + test("left anti early state exclusion on right") { + val (leftInput, rightInput, joined) = setupWindowedJoinWithRightCondition("left_anti") + + testStream(joined, OutputMode.Append())( + MultiAddData(leftInput, 3, 4, 5)(rightInput, 1, 2, 3), + // The right rows with rightValue <= 7 (i.e. right 1 and 2) fail the pre-join filter, so they + // are not added to the state and cannot suppress any left row. + CheckNewAnswer(), + // states + // left: 3, 4, 5 (3 matched right 3, so carries matched = true) + // right: 3 + assertNumStateRows( + total = Seq(4), updated = Seq(4), + droppedByWatermark = Seq(0), removed = Some(Seq(0))), + // Advance the watermark: left 4 and 5 were never matched, so they are emitted now. + MultiAddData(leftInput, 20)(rightInput, 21), + CheckNewAnswer(Row(4, 10, 8), Row(5, 10, 10)), + // states + // left: 20 + // right: 21 + // + // states evicted + // left: 3, 4, 5 (below watermark) + // right: 3 (below watermark) + // + // Only the single right side row is counted as removed - see the note in + // "windowed left anti join" about left side eviction not feeding numRemovedStateRows. + assertNumStateRows( + total = Seq(2), updated = Seq(2), + droppedByWatermark = Seq(0), removed = Some(Seq(1))) + ) + } + + test("left anti join is not supported in Update output mode") { + val (_, _, joined) = setupWindowedJoin("left_anti") + + val e = intercept[AnalysisException] { + joined.writeStream.format("memory").queryName("leftAntiUpdate") + .outputMode(OutputMode.Update()).start() + } + assert(e.getMessage.contains("LeftAnti join between two streaming DataFrames/Datasets " + + "is not supported in Update output mode, only in Append output mode")) + } +} + @SlowSQLTest class StreamingLeftSemiJoinWithVCFSuite extends StreamingLeftSemiJoinSuite { override protected def testMode = Mode.WithVCF @@ -2641,3 +2802,13 @@ class StreamingLeftSemiJoinWithVCFSuite extends StreamingLeftSemiJoinSuite { class StreamingLeftSemiJoinWithoutVCFSuite extends StreamingLeftSemiJoinSuite { override protected def testMode = Mode.WithoutVCF } + +@SlowSQLTest +class StreamingLeftAntiJoinWithVCFSuite extends StreamingLeftAntiJoinSuite { + override protected def testMode = Mode.WithVCF +} + +@SlowSQLTest +class StreamingLeftAntiJoinWithoutVCFSuite extends StreamingLeftAntiJoinSuite { + override protected def testMode = Mode.WithoutVCF +} From d5e7e6dad0ca7fa6fcafec0f04dedbcde9dad833 Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Thu, 6 Aug 2026 06:52:40 +0000 Subject: [PATCH 2/9] [SPARK-58611][SS][FOLLOWUP] Add state format v4 and range-condition anti join coverage * Runtime coverage missed state format v4. `skipUpdatingMatchedFlag` in `StreamingSymmetricHashJoinExec` is gated on `stateFormatVersion == 4`, so left anti takes a distinct path there, but the anti suites only covered v2/v3. Split `StreamingLeftAntiJoinSuite` into `StreamingLeftAntiJoinBase` plus a subclass holding the V1-V3-only tests -- mirroring the existing `StreamingLeftSemiJoinBase` / `StreamingLeftSemiJoinSuite` split -- and add `StreamingLeftAntiJoinV4Suite` alongside the other V4 suites so it inherits the runtime tests. * Range-condition joins use the state value watermark path rather than the state key watermark path exercised by the windowed anti tests. Add a `setupJoinWithRangeCondition("left_anti")` test covering it, and extend that helper's projection to treat `left_anti` like `left_semi` (left columns only). * The guide said an anti join "must specify watermark on right + time constraints", but the analyzer routes LeftAnti through the shared `checkForStreamStreamJoinWatermark`, so a watermarked column in the equality join keys on either side is also accepted. Document both ways of expressing the event-time constraint. No change to the implementation -- tests and documentation only. --- .../apis-on-dataframes-and-datasets.md | 14 ++++-- .../sql/streaming/StreamingJoinSuite.scala | 49 ++++++++++++++++++- .../sql/streaming/StreamingJoinV4Suite.scala | 5 ++ 3 files changed, 62 insertions(+), 6 deletions(-) diff --git a/docs/streaming/apis-on-dataframes-and-datasets.md b/docs/streaming/apis-on-dataframes-and-datasets.md index a133fc112ac9b..5c3236a165b99 100644 --- a/docs/streaming/apis-on-dataframes-and-datasets.md +++ b/docs/streaming/apis-on-dataframes-and-datasets.md @@ -1324,10 +1324,16 @@ regarding watermark delays and whether data will be dropped or not. ##### Anti Joins with Watermarking An anti join returns values from the left side of the relation that has no match with the right. -It is also referred to as a left anti join. As with semi joins, watermark + event-time constraints -must be specified for an anti join: since a row is emitted precisely because it has *no* match, the -engine has to wait until the watermark guarantees that no matching row can arrive on the right side -in future before it can emit the row. +It is also referred to as a left anti join. As with semi joins, watermarking and event-time +constraints must be specified for an anti join: since a row is emitted precisely because it has +*no* match, the engine has to wait until the watermark guarantees that no matching row can arrive +on the right side in future before it can emit the row. + +As for the other stateful join types, the event-time constraint can be expressed in either of two +ways: a watermarked event-time column can appear in the equality join keys, or a watermark can be +defined on the right side together with a time range condition (for example +`leftTime BETWEEN rightTime - INTERVAL 1 HOUR AND rightTime`). Defining a watermark on the left +side as well is optional, and is what allows the left side state to be cleaned up. Note that anti join is only supported in Append output mode. Update mode would have to emit rows early, before the watermark can rule out a future match, and such a row could be invalidated by a diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala index 4bf1a58bd639b..ead8e406e349f 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala @@ -262,7 +262,7 @@ abstract class StreamingJoinSuite s"leftTime BETWEEN rightTime - $lowerBound AND rightTime + $upperBound"), joinType) - val select = if (joinType == "left_semi") { + val select = if (joinType == "left_semi" || joinType == "left_anti") { joined.select($"leftKey", $"leftTime".cast("int")) } else { joined.select($"leftKey", $"rightKey", $"leftTime".cast("int"), @@ -2632,7 +2632,7 @@ class StreamingFullOuterJoinWithoutVCFSuite extends StreamingFullOuterJoinSuite override protected def testMode = Mode.WithoutVCF } -abstract class StreamingLeftAntiJoinSuite extends StreamingJoinSuite { +abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { test("windowed left anti join") { withTempDir { checkpointDir => @@ -2781,6 +2781,51 @@ abstract class StreamingLeftAntiJoinSuite extends StreamingJoinSuite { ) } + test("left anti join with watermark range condition") { + val (leftInput, rightInput, joined) = setupJoinWithRangeCondition("left_anti") + + testStream(joined, OutputMode.Append())( + AddData(leftInput, (1, 5), (3, 5)), + // Neither left row is provably unmatched yet. + CheckNewAnswer(), + // states + // left: (1, 5), (3, 5) + // right: nothing + assertNumStateRows( + total = Seq(2), updated = Seq(2), + droppedByWatermark = Seq(0), removed = Some(Seq(0))), + AddData(rightInput, (1, 10), (2, 5)), + // Right (1, 10) satisfies the range condition against left (1, 5), so that left row is + // marked as matched in state and must never be emitted. Unlike left semi, it is kept in + // state so that it can be suppressed at eviction time. + CheckNewAnswer(), + // states + // left: (1, 5) (now matched), (3, 5) + // right: (1, 10), (2, 5) + assertNumStateRows( + total = Seq(4), updated = Seq(2), + droppedByWatermark = Seq(0), removed = Some(Seq(0))), + // Advance the watermark to 20 by adding rows with event time 30 on both sides. A left row is + // only safe to evict once no future right row can match it, which the range condition puts + // at leftTime + 5 < watermark, so the left rows with leftTime < 15 are evicted here: + // (3, 5) was never matched so it is emitted, while (1, 5) was matched so it is suppressed. + AddData(leftInput, (1, 30)), + CheckNewAnswer(), + AddData(rightInput, (0, 30)), + CheckNewAnswer(Row(3, 5)), + // Advance the watermark to 50, which evicts the left rows with leftTime < 45. Left (1, 30) + // never matched -- the only right row with key 1 was evicted long before -- so it is + // emitted now. + AddData(leftInput, (2, 60)), + CheckNewAnswer(), + AddData(rightInput, (0, 60)), + CheckNewAnswer(Row(1, 30)) + ) + } +} + +abstract class StreamingLeftAntiJoinSuite extends StreamingLeftAntiJoinBase { + test("left anti join is not supported in Update output mode") { val (_, _, joined) = setupWindowedJoin("left_anti") diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinV4Suite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinV4Suite.scala index 6d4a97861efef..fa064e64fec6e 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinV4Suite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinV4Suite.scala @@ -476,3 +476,8 @@ class StreamingFullOuterJoinV4Suite class StreamingLeftSemiJoinV4Suite extends StreamingLeftSemiJoinBase with TestWithV4StateFormat + +@SlowSQLTest +class StreamingLeftAntiJoinV4Suite + extends StreamingLeftAntiJoinBase + with TestWithV4StateFormat From 772c64ef592ee5f68de6d90723c483e5cfc7c743 Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Mon, 10 Aug 2026 11:04:11 +0000 Subject: [PATCH 3/9] [SPARK-58611][SS][FOLLOWUP] Base left anti stream-stream join on left semi Rework the left anti stream-stream join to be a hybrid of left semi and left outer, as suggested in review, rather than a variant of left outer. Left anti is the mirror of left semi: a left row that finds any match can never be anti output, so it can be dropped from state on match instead of being kept with a `matched` flag until eviction. This reuses the left semi optimizations: skip storing a left row that matches on arrival, and remove an already-stored left row via `getJoinedRowsAndRemoveMatched` when a later right row matches it. The right side is processed first, as for left semi. Only the eviction-time emission of the surviving (never-matched) left rows stays left-outer-shaped. As a result the left-side `matched` flag is no longer consulted at eviction (every survivor is unmatched by construction), so its v4 `skipUpdatingMatchedFlag` special-casing is dropped. Row outputs are unchanged; state size for matched left rows now matches left semi rather than left outer. Update the affected `assertNumStateRows` expectations accordingly. --- .../join/StreamingSymmetricHashJoinExec.scala | 104 +++++++++--------- .../sql/streaming/StreamingJoinSuite.scala | 52 ++++----- 2 files changed, 80 insertions(+), 76 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala index de0e33edee84c..25028e41f3105 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala @@ -408,6 +408,8 @@ case class StreamingSymmetricHashJoinExec( // new left input with stored right input, and also stores all the left input. // - Left Semi Join: generates all new left input rows from matching new left input with // stored right input, and also stores all the non-matched left input. + // - Left Anti Join: like left semi, but generates nothing; it stores only the non-matched + // new left input (a matched left row can never be anti output, so it is not buffered). // // - `rightSideJoiner.storeAndJoinWithOtherSide(leftSideJoiner)` // - Inner, Left Outer, Right Outer, Full Outer Join: generates all rows from matching @@ -418,16 +420,18 @@ case class StreamingSymmetricHashJoinExec( // - Left Semi Join: generates all stored left input rows, from matching new right input // with stored left input, and also stores all the right input. Note only first-time // matched left input rows will be generated, this is to guarantee left semi semantics. + // - Left Anti Join: generates nothing, but removes the matched left rows from state (they + // can no longer become anti output), and also stores all the right input. // - // For Left Semi Join, we process the right side first so that new right input is stored - // before left input probes against it. This way, new left rows that match new right rows - // in the same microbatch are emitted immediately without being buffered in state. + // For Left Semi and Left Anti Join, we process the right side first so that new right input is + // stored before left input probes against it. This way, new left rows that match new right + // rows in the same microbatch are handled immediately without being buffered in state (emitted + // for left semi, dropped for left anti). // - // Left Anti Join generates nothing while joining, on either side: whether a left row has no - // match is only decided once the watermark guarantees no future right row can match it, so - // unmatched left rows are emitted when the left side state is evicted (see `outputIter`). Both - // sides still store their input, and matching a new right input marks the corresponding stored - // left rows as matched so that they are suppressed at eviction time. + // Left Anti differs from left semi only in what it emits: nothing during the join. A surviving + // (never-matched) left row is not provably unmatched until the watermark rules out any future + // right match, so -- like left outer -- those rows are emitted when the left state is evicted + // (see `outputIter`). val leftOutputIter = joinerManager.leftSideJoiner.storeAndJoinWithOtherSide(joinerManager.rightSideJoiner) { (input: InternalRow, matched: InternalRow) => joinedRow.withLeft(input).withRight(matched) @@ -447,10 +451,10 @@ case class StreamingSymmetricHashJoinExec( // this will be prepended to a second iterator producing other rows; for inner and left semi // joins, this is the full output. // - // For left semi join, right side is processed first so new right rows are available in - // state when left rows probe, avoiding unnecessary left-side state buffering. + // For left semi and left anti join, right side is processed first so new right rows are + // available in state when left rows probe, avoiding unnecessary left-side state buffering. val hashJoinOutputIter = CompletionIterator[InternalRow, Iterator[InternalRow]]( - if (joinType == LeftSemi) rightOutputIter ++ leftOutputIter + if (joinType == LeftSemi || joinType == LeftAnti) rightOutputIter ++ leftOutputIter else leftOutputIter ++ rightOutputIter, onHashJoinOutputCompletion()) @@ -549,22 +553,18 @@ case class StreamingSymmetricHashJoinExec( hashJoinOutputIter ++ leftSideOutputIter ++ rightSideOutputIter case LeftAnti => - // Left anti join is structurally an eviction-time join, like left outer: whether a left - // row has "no match" is only known once the watermark guarantees no future right row can - // match it. So we reuse the left outer eviction path, with two differences: - // * nothing is emitted when a left row matches, and - // * an unmatched left row is emitted as the bare left row rather than joined with nulls. + // Left anti join is a hybrid of left semi and left outer. The matched-row handling is the + // left semi optimization: a left row that finds any match can never be anti output, so it + // is dropped from state on match rather than kept (see `storeAndJoinWithOtherSide`), + // exactly mirroring left semi. As a result every left row that survives in state is, by + // construction, still unmatched. // - // State format version 1 is already rejected for non-inner joins (see the constructor), - // so only the 'matched' flag path needs to be handled. + // Whether such a surviving row is truly unmatched is only decided once the watermark + // guarantees no future right row can match it, so -- like the left side of a left outer + // join -- the survivors are emitted at eviction time, as bare left rows. Unlike left + // outer, no matched-flag filter is needed here, since matched rows are already gone. val initIterFn = { () => - val removedRowIter = joinerManager.leftSideJoiner.removeAndReturnOldState() - removedRowIter.filterNot { kv => - stateFormatVersion match { - case 2 | 3 | 4 => kv.matched - case _ => throwBadStateFormatVersionException() - } - }.map(_.value) + joinerManager.leftSideJoiner.removeAndReturnOldState().map(_.value) } // NOTE: we need to make sure `antiOutputIter` is evaluated "after" exhausting all of @@ -573,10 +573,10 @@ case class StreamingSymmetricHashJoinExec( // Please refer SPARK-38684 for more details. val antiOutputIter = new LazilyInitializingRowIterator(initIterFn) - // `hashJoinOutputIter` still has to be consumed: draining it is what appends input rows to - // the state stores and persists the 'matched' flag for left rows which found a match. For - // left anti it only carries the rows which failed the pre-join filter (see - // `generateFilteredJoinedRow`), which are genuine anti output, so it is concatenated as-is. + // `hashJoinOutputIter` still has to be consumed: draining it is what appends unmatched left + // input to state and removes newly-matched left rows. For left anti it only carries the + // rows which failed the pre-join filter (see `generateFilteredJoinedRow`), which are + // genuine anti output, so it is concatenated as-is. hashJoinOutputIter ++ antiOutputIter case _ => throwBadJoinTypeException() } @@ -605,8 +605,8 @@ case class StreamingSymmetricHashJoinExec( } // Count rows removed from the other side's state during the join phase - // (e.g., left semi join optimization removes matched left-side rows while processing - // right-side input, instead of a separate eviction pass). + // (e.g., the left semi / left anti join optimization removes matched left-side rows while + // processing right-side input, instead of a separate eviction pass). numRemovedStateRows += joinerManager.numRemovedFromOtherSideDuringJoin allRemovalsTimeMs += timeTakenMs { @@ -831,8 +831,8 @@ case class StreamingSymmetricHashJoinExec( // For older versions, we do not apply the optimization as it is a behavioral change, // although the optimization is valid for all versions. // - // Left anti join needs the flag on the left side rows only, so that unmatched left rows can - // be identified at eviction time. That flag is written while processing the right side. + // Left anti mirrors left semi here: matched left rows are removed from state on match, so no + // matched flag is consulted at eviction time (every survivor is unmatched by construction). val needToUpdateMatchedOnOtherSide = joinType match { case Inner => false case LeftOuter => joinSide == RightSide @@ -861,8 +861,11 @@ case class StreamingSymmetricHashJoinExec( case _ => (_: InternalRow, joinedRowIter: Iterator[JoinedRow]) => joinedRowIter } + // Both left semi and left anti remove a matched left row from state while processing the + // right side: once matched, a left row can no longer become semi output (already emitted) or + // anti output (permanently disqualified), so there is no reason to keep it. val removeMatchedFromOtherSideState = - joinType == LeftSemi && joinSide == RightSide + (joinType == LeftSemi || joinType == LeftAnti) && joinSide == RightSide nonLateRows.flatMap { row => val thisRow = row.asInstanceOf[UnsafeRow] @@ -872,9 +875,9 @@ case class StreamingSymmetricHashJoinExec( // the case of inner join). if (preJoinFilter(thisRow)) { val key = keyGenerator(thisRow) - // If the join type is Left Semi and this is the right side, we can remove the matched - // row from the other (left) side's state, since the row won't be produced anymore for - // the following input rows. + // If the join type is Left Semi or Left Anti and this is the right side, we can remove + // the matched row from the other (left) side's state, since the row won't be produced + // anymore for the following input rows. val joinedRowIter: Iterator[JoinedRow] = if (removeMatchedFromOtherSideState) { otherSideJoiner.joinStateManager.getJoinedRowsAndRemoveMatched( key, @@ -892,16 +895,17 @@ case class StreamingSymmetricHashJoinExec( skipUpdatingMatchedFlag) } if (joinType == LeftAnti) { - // Left anti join emits nothing when rows match, on either side; unmatched left rows - // are emitted later during watermark-based eviction of the left side state. The match + // Left anti join emits nothing while joining, on either side; unmatched left rows are + // emitted later during watermark-based eviction of the left side state. The match // status is passed explicitly, since it can no longer be inferred from the (empty) - // output iterator. + // output iterator: on the left side it drives skip-on-match (a matched left row is not + // stored), mirroring left semi. // - // The iterator must be drained fully rather than short-circuited on the first match: - // `getJoinedRows` sets the 'matched' flag on the other side's rows lazily, as they are - // produced, so stopping early would leave some matched left rows flagged as unmatched - // and they would then be wrongly emitted as anti output at eviction time. Draining is - // also what the state manager API requires of callers. + // The iterator must be drained fully rather than short-circuited on the first match. + // On the right side it is the `getJoinedRowsAndRemoveMatched` iterator, so draining is + // what removes the matched left rows from state; on the left side draining is required + // by the state manager API. Either way, draining is also needed to detect a match at + // all, since the anti output produces no rows to observe. var matched = false while (joinedRowIter.hasNext) { joinedRowIter.next() @@ -948,11 +952,11 @@ case class StreamingSymmetricHashJoinExec( // state store if it's right outer join or full outer join, as unmatched output is // handled during state eviction. // - // Note left anti deliberately does not get the left semi skip-on-match treatment: a matched - // left row has to stay in state (carrying matched = true) so that it is suppressed, rather - // than emitted, when the left side state is evicted. - val isLeftSemiWithMatch = joinType == LeftSemi && joinSide == LeftSide && iteratorNotEmpty - val shouldAddToState = if (isLeftSemiWithMatch) { + // Left anti gets the same left-side skip-on-match treatment as left semi: a matched left + // row can never be anti output, so there is no reason to keep it in state. + val isLeftSemiOrAntiWithMatch = + (joinType == LeftSemi || joinType == LeftAnti) && joinSide == LeftSide && iteratorNotEmpty + val shouldAddToState = if (isLeftSemiOrAntiWithMatch) { false } else if (joinSide == LeftSide) { true diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala index ead8e406e349f..44e155dfd26c9 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala @@ -2645,21 +2645,21 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { // out a future match for them. Left 3, 4, 5 matched and are suppressed forever. CheckNewAnswer(), // states - // left: 1, 2, 3, 4, 5 (all buffered; 3, 4, 5 carry matched = true) + // left: 1, 2 (3, 4, 5 matched right rows, so they are not buffered -- see left semi) // right: 3, 4, 5, 6, 7 assertNumStateRows( - total = Seq(10), updated = Seq(10), + total = Seq(7), updated = Seq(7), droppedByWatermark = Seq(0), removed = Some(Seq(0))), MultiAddData(leftInput, 21)(rightInput, 22), - // Watermark = 11, so window=[0,10] is evicted from the left side: the unmatched left rows - // 1 and 2 are emitted now, while the matched 3, 4, 5 are dropped without output. + // Watermark = 11, so window=[0,10] is evicted from the left side: the surviving unmatched + // left rows 1 and 2 are emitted now (3, 4, 5 were never buffered, having matched). CheckNewAnswer(Row(1, 10, 2), Row(2, 10, 4)), // states // left: 21 // right: 22 // // states evicted - // left: 1, 2, 3, 4, 5 (below watermark) + // left: 1, 2 (below watermark) // right: 3, 4, 5, 6, 7 (below watermark) // // Only the 5 right side rows are reported as removed: the left side rows are evicted @@ -2672,13 +2672,13 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { // Restart the join query from the same checkpoint StartStream(checkpointLocation = checkpointDir.getCanonicalPath), AddData(leftInput, 22), - // Left 22 matches right 22, so it is buffered with matched = true and never emitted. + // Left 22 matches right 22, so it is not buffered and never emitted (as in left semi). CheckNewAnswer(), // states - // left: 21, 22 + // left: 21 // right: 22 assertNumStateRows( - total = Seq(3), updated = Seq(1), + total = Seq(2), updated = Seq(0), droppedByWatermark = Seq(0), removed = Some(Seq(0))), StopStream, // Restart the query from the same checkpoint @@ -2687,10 +2687,10 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { // Row not added as 1 < state key watermark = 12. CheckNewAnswer(), // states - // left: 21, 22 + // left: 21 // right: 22 assertNumStateRows( - total = Seq(3), updated = Seq(0), + total = Seq(2), updated = Seq(0), droppedByWatermark = Seq(1), removed = Some(Seq(0))) ) } @@ -2724,10 +2724,10 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { // never match and are emitted immediately without being added to the state. CheckNewAnswer(Row(1, 10, 2), Row(2, 10, 4)), // states - // left: 3 (matched right 3 in the same batch, buffered with matched = true) + // left: empty (3 matched right 3 in the same batch, so it is not buffered) // right: 3, 4, 5 assertNumStateRows( - total = Seq(4), updated = Seq(4), + total = Seq(3), updated = Seq(3), droppedByWatermark = Seq(0), removed = Some(Seq(0))), // Left 3 matched, so advancing the watermark must not produce an anti row for it. MultiAddData(leftInput, 20)(rightInput, 21), @@ -2737,11 +2737,10 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { // right: 21 // // states evicted - // left: 3 (below watermark, matched so no output) + // left: nothing (3 was never buffered, having matched) // right: 3, 4, 5 (below watermark) // - // Only the 3 right side rows are counted as removed - see the note in - // "windowed left anti join" about left side eviction not feeding numRemovedStateRows. + // Only the 3 right side rows are counted as removed. assertNumStateRows( total = Seq(2), updated = Seq(2), droppedByWatermark = Seq(0), removed = Some(Seq(3))) @@ -2757,10 +2756,10 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { // are not added to the state and cannot suppress any left row. CheckNewAnswer(), // states - // left: 3, 4, 5 (3 matched right 3, so carries matched = true) + // left: 4, 5 (3 matched right 3, so it is not buffered) // right: 3 assertNumStateRows( - total = Seq(4), updated = Seq(4), + total = Seq(3), updated = Seq(3), droppedByWatermark = Seq(0), removed = Some(Seq(0))), // Advance the watermark: left 4 and 5 were never matched, so they are emitted now. MultiAddData(leftInput, 20)(rightInput, 21), @@ -2770,11 +2769,12 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { // right: 21 // // states evicted - // left: 3, 4, 5 (below watermark) + // left: 4, 5 (below watermark) // right: 3 (below watermark) // - // Only the single right side row is counted as removed - see the note in - // "windowed left anti join" about left side eviction not feeding numRemovedStateRows. + // Only the single right side row is counted as removed - the left side rows are evicted + // through removeAndReturnOldState while generating the anti output, which does not feed + // numRemovedStateRows. assertNumStateRows( total = Seq(2), updated = Seq(2), droppedByWatermark = Seq(0), removed = Some(Seq(1))) @@ -2796,19 +2796,19 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { droppedByWatermark = Seq(0), removed = Some(Seq(0))), AddData(rightInput, (1, 10), (2, 5)), // Right (1, 10) satisfies the range condition against left (1, 5), so that left row is - // marked as matched in state and must never be emitted. Unlike left semi, it is kept in - // state so that it can be suppressed at eviction time. + // removed from state and must never be emitted, exactly as in left semi. CheckNewAnswer(), // states - // left: (1, 5) (now matched), (3, 5) + // left: (3, 5) ((1, 5) removed on match) // right: (1, 10), (2, 5) assertNumStateRows( - total = Seq(4), updated = Seq(2), - droppedByWatermark = Seq(0), removed = Some(Seq(0))), + total = Seq(3), updated = Seq(2), + droppedByWatermark = Seq(0), removed = Some(Seq(1))), // Advance the watermark to 20 by adding rows with event time 30 on both sides. A left row is // only safe to evict once no future right row can match it, which the range condition puts // at leftTime + 5 < watermark, so the left rows with leftTime < 15 are evicted here: - // (3, 5) was never matched so it is emitted, while (1, 5) was matched so it is suppressed. + // (3, 5) was never matched so it is emitted. (1, 5) is not among them -- it was already + // removed from state when it matched in the previous batch. AddData(leftInput, (1, 30)), CheckNewAnswer(), AddData(rightInput, (0, 30)), From ab181ec60f2f9529fdf2a715bdafe9c5f21f7809 Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Mon, 10 Aug 2026 17:45:56 +0000 Subject: [PATCH 4/9] [SPARK-58611][SS][FOLLOWUP] Fix left anti join docs watermark and examples Correct the watermark/state-cleanup explanation in the anti join section: for left anti it is the right side watermark that lets the engine decide a left row can no longer match and emit it, so the right watermark drives eviction and output of left side state, while the optional left watermark is what allows the right side state to be cleaned up. The previous text had this reversed. Also add leftAnti / left_anti to the supported join type comments in the stream-stream join example snippets across all language tabs. --- .../apis-on-dataframes-and-datasets.md | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/streaming/apis-on-dataframes-and-datasets.md b/docs/streaming/apis-on-dataframes-and-datasets.md index 5c3236a165b99..6d3aec6b7753f 100644 --- a/docs/streaming/apis-on-dataframes-and-datasets.md +++ b/docs/streaming/apis-on-dataframes-and-datasets.md @@ -1230,7 +1230,7 @@ impressionsWithWatermark.join( clickTime >= impressionTime AND clickTime <= impressionTime + interval 1 hour """), - "leftOuter" # can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi" + "leftOuter" # can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" ) {% endhighlight %} @@ -1248,7 +1248,7 @@ impressionsWithWatermark.join( clickTime >= impressionTime AND clickTime <= impressionTime + interval 1 hour """), - joinType = "leftOuter" // can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi" + joinType = "leftOuter" // "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" ) {% endhighlight %} @@ -1264,7 +1264,7 @@ impressionsWithWatermark.join( "clickAdId = impressionAdId AND " + "clickTime >= impressionTime AND " + "clickTime <= impressionTime + interval 1 hour "), - "leftOuter" // can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi" + "leftOuter" // can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" ); {% endhighlight %} @@ -1283,7 +1283,7 @@ joined <- join( "clickAdId = impressionAdId AND", "clickTime >= impressionTime AND", "clickTime <= impressionTime + interval 1 hour"), - "left_outer" # can be "inner", "left_outer", "right_outer", "full_outer", "left_semi" + "left_outer" # "inner", "left_outer", "right_outer", "full_outer", "left_semi", "left_anti" )) {% endhighlight %} @@ -1332,8 +1332,10 @@ on the right side in future before it can emit the row. As for the other stateful join types, the event-time constraint can be expressed in either of two ways: a watermarked event-time column can appear in the equality join keys, or a watermark can be defined on the right side together with a time range condition (for example -`leftTime BETWEEN rightTime - INTERVAL 1 HOUR AND rightTime`). Defining a watermark on the left -side as well is optional, and is what allows the left side state to be cleaned up. +`leftTime BETWEEN rightTime - INTERVAL 1 HOUR AND rightTime`). The right side watermark is what +lets the engine decide that a left row can no longer be matched and emit it, so it drives the +eviction and output of left side state. Defining a watermark on the left side as well is optional, +and is what allows the right side state to be cleaned up. Note that anti join is only supported in Append output mode. Update mode would have to emit rows early, before the watermark can rule out a future match, and such a row could be invalidated by a @@ -1549,7 +1551,7 @@ joined = impressionsWithWatermark.join( clickTime >= impressionTime AND clickTime <= impressionTime + interval 1 hour """), - "leftOuter" # can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi" + "leftOuter" # can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" ) joined.groupBy( @@ -1571,7 +1573,7 @@ val joined = impressionsWithWatermark.join( clickTime >= impressionTime AND clickTime <= impressionTime + interval 1 hour """), - joinType = "leftOuter" // can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi" + joinType = "leftOuter" // "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" ) joined @@ -1590,7 +1592,7 @@ Dataset joined = impressionsWithWatermark.join( "clickAdId = impressionAdId AND " + "clickTime >= impressionTime AND " + "clickTime <= impressionTime + interval 1 hour "), - "leftOuter" // can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi" + "leftOuter" // can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" ); joined From e85f3b14f40163e3cbb7d9bc4b1460b4fd47fc28 Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Mon, 17 Aug 2026 16:16:18 +0000 Subject: [PATCH 5/9] [SPARK-58611][SS][TESTS] Rename misleading left anti join test The test asserted empty output throughout (the left row gets matched), so its "emits an unmatched left row" name was misleading. Rename it to describe what it verifies and note the positive emission case is covered by "windowed left anti join". --- .../org/apache/spark/sql/streaming/StreamingJoinSuite.scala | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala index 44e155dfd26c9..5b0597d2906b5 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala @@ -2696,12 +2696,14 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { } } - test("left anti join emits an unmatched left row only once the watermark passes it") { + test("left anti join does not emit a left row matched by a right row in a later batch") { val (leftInput, rightInput, joined) = setupWindowedJoin("left_anti") testStream(joined, OutputMode.Append())( AddData(leftInput, 3), - // Unmatched so far, but not yet provably unmatched. + // Unmatched so far, but not yet provably unmatched, so nothing is emitted. This is the + // window in which a match can still arrive and cancel the eventual anti output -- the + // positive "emitted once the watermark passes" case is covered by "windowed left anti join". CheckNewAnswer(), // A matching right row arrives in a later batch, so left 3 must never be emitted. AddData(rightInput, 3), From 893d3f5fd78dbfff69764b8c3495cddd4f2f4b19 Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Tue, 18 Aug 2026 12:08:59 +0000 Subject: [PATCH 6/9] [SPARK-58611][SS][FOLLOWUP] Require correct watermark placement for left anti stream-stream join A left anti stream-stream join emits its output (unmatched left rows) from left-state eviction, so it needs both the left state evicted and the right side late-filtered on the matching dimension. The generic watermark check guaranteed neither, so the analyzer accepted configs that silently produced wrong output, e.g. a right-watermark-only range join (left state never evicted). Tighten the LeftAnti analyzer check: the range path requires a left-side watermark; the equi-join path requires the watermark on the right join key at the eviction ordinal (new StreamingJoinHelper.isWatermarkOnRightEvictionJoinKey). Left outer/semi are unchanged, pending a separate config-gated fix. Also fix the docs and add analyzer/runtime coverage. --- .../apis-on-dataframes-and-datasets.md | 49 ++++++--- .../analysis/StreamingJoinHelper.scala | 32 ++++++ .../UnsupportedOperationChecker.scala | 50 +++++++++ .../analysis/UnsupportedOperationsSuite.scala | 102 +++++++++++++++++- .../join/StreamingSymmetricHashJoinExec.scala | 15 +-- .../sql/streaming/StreamingJoinSuite.scala | 24 ++++- 6 files changed, 244 insertions(+), 28 deletions(-) diff --git a/docs/streaming/apis-on-dataframes-and-datasets.md b/docs/streaming/apis-on-dataframes-and-datasets.md index 6d3aec6b7753f..89eca8af83291 100644 --- a/docs/streaming/apis-on-dataframes-and-datasets.md +++ b/docs/streaming/apis-on-dataframes-and-datasets.md @@ -1326,16 +1326,24 @@ regarding watermark delays and whether data will be dropped or not. An anti join returns values from the left side of the relation that has no match with the right. It is also referred to as a left anti join. As with semi joins, watermarking and event-time constraints must be specified for an anti join: since a row is emitted precisely because it has -*no* match, the engine has to wait until the watermark guarantees that no matching row can arrive -on the right side in future before it can emit the row. - -As for the other stateful join types, the event-time constraint can be expressed in either of two -ways: a watermarked event-time column can appear in the equality join keys, or a watermark can be -defined on the right side together with a time range condition (for example -`leftTime BETWEEN rightTime - INTERVAL 1 HOUR AND rightTime`). The right side watermark is what -lets the engine decide that a left row can no longer be matched and emit it, so it drives the -eviction and output of left side state. Defining a watermark on the left side as well is optional, -and is what allows the right side state to be cleaned up. +*no* match, the engine generally has to wait until the watermark guarantees that no matching row +can arrive on the right side in future before it can emit the row. (The exception is a left row +that fails a deterministic left-side-only predicate in the join condition: it can never match any +right row, so it is emitted immediately without waiting for the watermark.) + +Unlike outer joins, an anti join has stricter watermark requirements, because two independent things +must both hold. First, the right side must be late-filtered on the matching dimension, so that a +right row arriving too late to matter is dropped rather than processed; without this a late right +row could match a left row that has *already* been emitted as an anti row, silently corrupting the +result. Second, the left state must be evicted, since eviction is what emits the surviving unmatched +left rows. Concretely: for an equality join on a watermarked event-time key, that key must be +watermarked on the **right** side (the left state is then evicted through the shared key); for a +time range condition (for example `leftTime BETWEEN rightTime - INTERVAL 1 HOUR AND rightTime`), a +watermark must be defined on **both** sides -- the right side for late filtering and the left side +for eviction. The recommended, always-correct configuration is to watermark both sides on the +event-time column used by the join. Configurations that leave the right side un-filtered (for +example a watermark on only the left join key) or the left state never evicted (a watermark on only +the right side of a range condition) are rejected at analysis time. Note that anti join is only supported in Append output mode. Update mode would have to emit rows early, before the watermark can rule out a future match, and such a row could be invalidated by a @@ -1343,8 +1351,9 @@ later batch. ###### Semantic Guarantees of Stream-stream Anti Joins with Watermarking Anti joins have the same guarantees regarding watermark delays and whether data will be dropped as -[outer joins](#outer-joins-with-watermarking), because unmatched rows are likewise only emitted once -the watermark has passed them. +[outer joins](#outer-joins-with-watermarking), because surviving unmatched rows are likewise only +emitted once the watermark has passed them (the sole exception being left rows that fail a +deterministic left-side-only predicate, which are emitted immediately as noted above). ##### Support matrix for joins in streaming queries @@ -1458,8 +1467,9 @@ the watermark has passed them. @@ -1551,7 +1561,8 @@ joined = impressionsWithWatermark.join( clickTime >= impressionTime AND clickTime <= impressionTime + interval 1 hour """), - "leftOuter" # can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" + "leftOuter" # "inner", "leftOuter", "rightOuter", "fullOuter" (not "leftSemi"/"leftAnti": + # they output left columns only, which this aggregation on click* does not have) ) joined.groupBy( @@ -1573,7 +1584,9 @@ val joined = impressionsWithWatermark.join( clickTime >= impressionTime AND clickTime <= impressionTime + interval 1 hour """), - joinType = "leftOuter" // "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" + // "inner", "leftOuter", "rightOuter", "fullOuter" (not "leftSemi"/"leftAnti": they output left + // columns only, which this aggregation on click* does not have) + joinType = "leftOuter" ) joined @@ -1592,7 +1605,9 @@ Dataset joined = impressionsWithWatermark.join( "clickAdId = impressionAdId AND " + "clickTime >= impressionTime AND " + "clickTime <= impressionTime + interval 1 hour "), - "leftOuter" // can be "inner", "leftOuter", "rightOuter", "fullOuter", "leftSemi", "leftAnti" + // "inner", "leftOuter", "rightOuter", "fullOuter" (not "leftSemi"/"leftAnti": they output left + // columns only, which this aggregation on click* does not have) + "leftOuter" ); joined diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala index c4549a189e8e1..929102745c143 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala @@ -51,6 +51,38 @@ object StreamingJoinHelper extends PredicateHelper with Logging { } } + /** + * Whether the watermark sits on the right equality join key at the ordinal that actually drives + * state-key eviction. This is stricter than [[isWatermarkInJoinKeys]] (side-insensitive) and than + * simply "some right key is watermarked" (ordinal-insensitive). It matters for join types (e.g. + * left anti) whose correctness depends on the right side being late-filtered on the *same* key + * dimension the left state is evicted by. + * + * State-key eviction uses a single key ordinal, chosen exactly as in + * [[org.apache.spark.sql.execution.streaming.operators.stateful.join. + * StreamingSymmetricHashJoinHelper.findJoinKeyOrdinalForWatermark]]: the first watermarked left + * key, else the first watermarked right key. The right side is late-filtered on its own + * watermarked column, so unless the right key at that ordinal is the watermarked one, a right row + * that is old on the eviction key but fresh on some other watermarked key/column would not be + * late-filtered and could match an already-emitted anti row. This method must stay in sync with + * that ordinal selection. + */ + def isWatermarkOnRightEvictionJoinKey(plan: LogicalPlan): Boolean = { + plan match { + case ExtractEquiJoinKeys(_, leftKeys, rightKeys, _, _, _, _, _) => + def watermarked(e: Expression): Boolean = e match { + case ne: NamedExpression => ne.metadata.contains(EventTimeWatermark.delayKey) + case _ => false + } + val ordinal = leftKeys.indexWhere(watermarked) match { + case i if i >= 0 => i + case _ => rightKeys.indexWhere(watermarked) + } + ordinal >= 0 && ordinal < rightKeys.length && watermarked(rightKeys(ordinal)) + case _ => false + } + } + /** * Get state value watermark (see [[StreamingSymmetricHashJoinExec]] for context about it) * given the join condition and the event time watermark. This is how it works. diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala index 363cb10b75f10..21891823c1e81 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala @@ -718,5 +718,55 @@ object UnsupportedOperationChecker extends Logging { "is not supported without a watermark in the join keys, or a watermark on " + "the nullable side and an appropriate range condition")(join) } + + // Left anti has stricter watermark requirements than the other stream-stream joins, because it + // is the only outer-like join whose output (the unmatched left rows) is produced from left + // state eviction AND can be invalidated by a *later* right row. Correct anti output needs both + // of the following, which the generic side-insensitive check above does not guarantee: + // + // 1. The right side must be late-filtered on the matching dimension, so a right row old + // enough to match an already-evicted (hence already-emitted) left row is dropped rather + // than processed. The operator late-filters a side using that side's own watermarked + // event-time column, so this bounds matching only when the watermark is on the right + // *join key* (equi-join path) or the right *range-bound* attribute (range path). + // 2. The left state must actually be evicted, so surviving unmatched left rows are emitted. + // A watermark in the join keys evicts the left key state; a range condition needs a + // watermark on the LEFT side to evict the left value state. + // + // Left outer/semi keep their existing one-sided-watermark behavior for compatibility; a general + // fix for them is tracked separately. Note the range path still only checks that the left side + // is watermarked, not that the watermark is on the range-bound attribute -- that column-level + // precision is a pre-existing gap shared by all stream-stream joins and is deferred. + if (join.joinType == LeftAnti) { + if (watermarkInJoinKeys) { + // Equi-join path: the left key state is evicted via the join-key watermark at a single + // ordinal, so we need the right side late-filtered on that *same* key ordinal. That + // requires the right key at the eviction ordinal to be watermarked -- a watermark on only + // the left key, on an unrelated right column, or on a different key position (composite + // keys) would let a late right row match an already-emitted anti row. + if (!StreamingJoinHelper.isWatermarkOnRightEvictionJoinKey(join)) { + throwError( + "Stream-stream LeftAnti join between two streaming DataFrame/Datasets with a " + + "watermarked join key requires the watermark to be on the right join key used for " + + "state eviction (for a composite key, the right key at the same position as the " + + "watermarked left key), so that late right rows are dropped and cannot invalidate " + + "already-emitted anti rows. A watermark on only the left key, on an unrelated " + + "right column, or on a different key position, is not supported.")(join) + } + } else { + // Range-condition path (hasValidWatermarkRange holds here): the right side is late-filtered + // via its range-bound watermark, but the left state is evicted -- and hence anti rows + // emitted -- only when the LEFT side is watermarked. + val leftHasWatermark = + join.left.output.exists(_.metadata.contains(EventTimeWatermark.delayKey)) + if (!leftHasWatermark) { + throwError( + "Stream-stream LeftAnti join between two streaming DataFrame/Datasets with a range " + + "condition requires a watermark on the left side, so that the left state (whose " + + "surviving unmatched rows form the anti output) is evicted. A watermark on only " + + "the right side is not supported.")(join) + } + } + } } } diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala index ae114e76f0aa9..c1c549f5a03f2 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala @@ -481,8 +481,10 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { Seq("is not supported in Complete output mode", allowedModesMsg)) } - // Left outer, right outer, full outer, left semi, left anti joins - Seq(LeftOuter, RightOuter, FullOuter, LeftSemi, LeftAnti).foreach { joinType => + // Left outer, right outer, full outer, left semi joins. + // Left anti is handled separately below: it has stricter watermark requirements (see + // checkForStreamStreamJoinWatermark), so the side-insensitive cases here do not all apply to it. + Seq(LeftOuter, RightOuter, FullOuter, LeftSemi).foreach { joinType => // Stream-stream allowed with join on watermark attribute // Note that the attribute need not be watermarked on both sides. assertSupportedInStreamingPlan( @@ -529,6 +531,102 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { "the nullable side and an appropriate range condition")) } + // Left anti join: stricter watermark requirements than the other join types. Correct anti output + // needs (1) the right side late-filtered on the matching dimension -- the right join key on the + // equi-join path, or the right range-bound attribute on the range path -- so a late right row + // cannot invalidate an already-emitted anti row, and (2) left-state eviction, via a join-key + // watermark or a watermark on the left side. Distinct attributes are used per side to avoid + // exprId collisions in these synthetic plans. + { + val leftWm = AttributeReference("lw", IntegerType)().withMetadata(watermarkMetadata) + val rightWm = AttributeReference("rw", IntegerType)().withMetadata(watermarkMetadata) + val leftPlain = AttributeReference("lp", IntegerType)() + val rightPlain = AttributeReference("rp", IntegerType)() + val rightOtherWm = AttributeReference("ro", IntegerType)().withMetadata(watermarkMetadata) + + // Supported: equality join key watermarked on the right (the left key state is evicted via the + // shared key watermark, and the right side is late-filtered on the join key). + assertSupportedInStreamingPlan( + "left anti join with stream-stream relations and join key watermarked on the right side", + new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightWm), + joinType = LeftAnti, condition = Some(leftPlain === rightWm)), + OutputMode.Append()) + + // Supported: range condition with a watermark on both sides. + assertSupportedInStreamingPlan( + "left anti join with stream-stream relations and range condition, watermark on both sides", + new TestStreamingRelation(leftWm).join(new TestStreamingRelation(rightWm), + joinType = LeftAnti, condition = Some(leftWm > rightWm + 10)), + OutputMode.Append()) + + // Not supported: equality join key watermarked on the left only -- the right join key is not + // late-filtered, so a late right row could invalidate an already-emitted anti row. + assertNotSupportedInStreamingPlan( + "left anti join with stream-stream relations and join key watermarked on the left side only", + new TestStreamingRelation(leftWm).join(new TestStreamingRelation(rightPlain), + joinType = LeftAnti, condition = Some(leftWm === rightPlain)), + OutputMode.Append(), + Seq("requires the watermark to be on the right join key")) + + // Not supported: the watermark is on an unrelated right column, not the right join key, so the + // right side is not late-filtered on the key that bounds matching. + assertNotSupportedInStreamingPlan( + "left anti join with stream-stream relations and watermark on an unrelated right column", + new TestStreamingRelation(leftWm).join( + new TestStreamingRelation(Seq(rightPlain, rightOtherWm)), + joinType = LeftAnti, condition = Some(leftWm === rightPlain)), + OutputMode.Append(), + Seq("requires the watermark to be on the right join key")) + + // Composite equality keys. State-key eviction uses a single ordinal (the first watermarked + // left key), so the right key at that ordinal must be the watermarked one. + val leftWm1 = AttributeReference("lw1", IntegerType)().withMetadata(watermarkMetadata) + val leftPlain2 = AttributeReference("lp2", IntegerType)() + val rightWm1 = AttributeReference("rw1", IntegerType)().withMetadata(watermarkMetadata) + val rightPlain1 = AttributeReference("rp1", IntegerType)() + val rightPlain2 = AttributeReference("rp2", IntegerType)() + val rightWm2 = AttributeReference("rw2", IntegerType)().withMetadata(watermarkMetadata) + + // Supported: composite key watermarked at the same ordinal on both sides (ordinal 0). + assertSupportedInStreamingPlan( + "left anti join with stream-stream relations and composite key watermarked at same ordinal", + new TestStreamingRelation(Seq(leftWm1, leftPlain2)).join( + new TestStreamingRelation(Seq(rightWm1, rightPlain2)), + joinType = LeftAnti, + condition = Some(leftWm1 === rightWm1 && leftPlain2 === rightPlain2)), + OutputMode.Append()) + + // Not supported: composite key watermarked on mismatched ordinals -- left key 0 and right + // key 1 are watermarked, but eviction uses ordinal 0, whose right key is not watermarked, so + // the right side is not late-filtered on the eviction key. + assertNotSupportedInStreamingPlan( + "left anti join with stream-stream relations and composite key watermarked on mismatched " + + "ordinals", + new TestStreamingRelation(Seq(leftWm1, leftPlain2)).join( + new TestStreamingRelation(Seq(rightPlain1, rightWm2)), + joinType = LeftAnti, + condition = Some(leftWm1 === rightPlain1 && leftPlain2 === rightWm2)), + OutputMode.Append(), + Seq("requires the watermark to be on the right join key")) + + // Not supported: range condition with a watermark on the right only -- the left state is never + // evicted, so no anti row is ever produced. + assertNotSupportedInStreamingPlan( + "left anti join with stream-stream relations and range condition, right watermark only", + new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightWm), + joinType = LeftAnti, condition = Some(leftPlain > rightWm + 10)), + OutputMode.Append(), + Seq("requires a watermark on the left side")) + + // Not supported: no watermark at all (rejected by the generic stream-stream join check). + assertNotSupportedInStreamingPlan( + "left anti join with stream-stream relations and no watermark", + new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightPlain), + joinType = LeftAnti, condition = Some(leftPlain === rightPlain)), + OutputMode.Append(), + Seq("without a watermark in the join keys")) + } + // multi-aggregations only supported in Append mode assertPassOnGlobalWatermarkLimit( "aggregate - multiple streaming aggregations - append", diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala index 25028e41f3105..083789b10e1cf 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala @@ -408,8 +408,10 @@ case class StreamingSymmetricHashJoinExec( // new left input with stored right input, and also stores all the left input. // - Left Semi Join: generates all new left input rows from matching new left input with // stored right input, and also stores all the non-matched left input. - // - Left Anti Join: like left semi, but generates nothing; it stores only the non-matched - // new left input (a matched left row can never be anti output, so it is not buffered). + // - Left Anti Join: like left semi, but generates nothing while joining; it stores only the + // non-matched new left input (a matched left row can never be anti output, so it is not + // buffered). A left row that fails the pre-join filter is unmatched by definition and is + // emitted immediately without being stored (see `generateFilteredJoinedRow`). // // - `rightSideJoiner.storeAndJoinWithOtherSide(leftSideJoiner)` // - Inner, Left Outer, Right Outer, Full Outer Join: generates all rows from matching @@ -428,10 +430,11 @@ case class StreamingSymmetricHashJoinExec( // rows in the same microbatch are handled immediately without being buffered in state (emitted // for left semi, dropped for left anti). // - // Left Anti differs from left semi only in what it emits: nothing during the join. A surviving - // (never-matched) left row is not provably unmatched until the watermark rules out any future - // right match, so -- like left outer -- those rows are emitted when the left state is evicted - // (see `outputIter`). + // Left Anti differs from left semi only in what it emits: for a row that passes the pre-join + // filter, nothing during the join (a left row that fails the pre-join filter is emitted + // immediately, since it can never match). A surviving (never-matched) left row is not provably + // unmatched until the watermark rules out any future right match, so -- like left outer -- + // those rows are emitted when the left state is evicted (see `outputIter`). val leftOutputIter = joinerManager.leftSideJoiner.storeAndJoinWithOtherSide(joinerManager.rightSideJoiner) { (input: InternalRow, matched: InternalRow) => joinedRow.withLeft(input).withRight(matched) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala index 5b0597d2906b5..636270a0df8ff 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala @@ -239,16 +239,20 @@ abstract class StreamingJoinSuite joinType: String, watermark: String = "10 seconds", lowerBound: String = "interval 5 seconds", - upperBound: String = "interval 5 seconds") + upperBound: String = "interval 5 seconds", + leftWatermark: Boolean = true) : (MemoryStream[(Int, Int)], MemoryStream[(Int, Int)], DataFrame) = { val leftInput = MemoryStream[(Int, Int)] val rightInput = MemoryStream[(Int, Int)] - val df1 = leftInput.toDF().toDF("leftKey", "time") + val df1Base = leftInput.toDF().toDF("leftKey", "time") .select($"leftKey", timestamp_seconds($"time") as "leftTime", ($"leftKey" * 2) as "leftValue") - .withWatermark("leftTime", watermark) + // The left watermark is optional for most join types; left anti requires it (a watermark on + // only the right side leaves the left state unevicted). Allow tests to omit it to exercise + // that path. + val df1 = if (leftWatermark) df1Base.withWatermark("leftTime", watermark) else df1Base val df2 = rightInput.toDF().toDF("rightKey", "time") .select($"rightKey", timestamp_seconds($"time") as "rightTime", @@ -2838,6 +2842,20 @@ abstract class StreamingLeftAntiJoinSuite extends StreamingLeftAntiJoinBase { assert(e.getMessage.contains("LeftAnti join between two streaming DataFrames/Datasets " + "is not supported in Update output mode, only in Append output mode")) } + + test("left anti join with a range condition requires a watermark on the left side") { + // Only the right side is watermarked. With a range condition (no watermark in the join keys), + // the operator would never build a left-side eviction predicate, so no anti row would ever be + // emitted. This must be rejected at analysis rather than silently producing no output. + val (_, _, joined) = setupJoinWithRangeCondition("left_anti", leftWatermark = false) + + val e = intercept[AnalysisException] { + joined.writeStream.format("memory").queryName("leftAntiRightWatermarkOnly") + .outputMode(OutputMode.Append()).start() + } + assert(e.getMessage.contains( + "requires a watermark on the left side")) + } } @SlowSQLTest From 1fe2ebc846d8ebc420eb52f6a1a66734b8f1c7d8 Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Wed, 19 Aug 2026 04:21:44 +0000 Subject: [PATCH 7/9] [SPARK-58611][SS][FOLLOWUP] Share and strengthen the watermark-placement check across left semi/outer/anti Generalize the LeftAnti-only watermark-placement check into a shared check for left semi/outer/anti. The equi-join path now requires both eviction join keys to be watermarked, and the range path requires the state watermark to derive from watermarked attributes on both sides. Left semi/outer are gated by the new config spark.sql.streaming.join.stricterWatermarkRequirements.enabled (default true, set false to restore the old behavior); left anti always enforces it. Also fixes the same-batch (SPARK-49829 split) case for left semi/anti: a non-late right row that is evicting in the batch is now stored so later same-batch left rows can match it, and the matched-row removal probe prunes the V4 state scan by timestamp. Updates the join docs, support matrix, and migration guide. --- .../apis-on-dataframes-and-datasets.md | 71 +++-- docs/streaming/ss-migration-guide.md | 4 + .../analysis/StreamingJoinHelper.scala | 75 ++++-- .../UnsupportedOperationChecker.scala | 141 +++++++--- .../apache/spark/sql/internal/SQLConf.scala | 18 ++ .../analysis/UnsupportedOperationsSuite.scala | 252 ++++++++++++++++-- .../join/StreamingSymmetricHashJoinExec.scala | 20 +- .../join/SymmetricHashJoinStateManager.scala | 25 +- .../SymmetricHashJoinStateManagerSuite.scala | 35 +++ .../sql/streaming/StreamingJoinSuite.scala | 166 +++++++++++- 10 files changed, 674 insertions(+), 133 deletions(-) diff --git a/docs/streaming/apis-on-dataframes-and-datasets.md b/docs/streaming/apis-on-dataframes-and-datasets.md index 89eca8af83291..79cda2115a43c 100644 --- a/docs/streaming/apis-on-dataframes-and-datasets.md +++ b/docs/streaming/apis-on-dataframes-and-datasets.md @@ -1293,6 +1293,20 @@ joined <- join( +For a **left outer** join, the surviving unmatched left rows are emitted when the left-side state is +evicted, and an already-emitted `NULL`-extended row can be invalidated by a right row that arrives +later. Correct results therefore require the watermark to be placed so that (1) the left state is +actually evicted and (2) both sides are late-filtered on the dimension that bounds matching. +Concretely: for an equality join on a watermarked event-time key, that key must be watermarked on +**both** sides; for a time range condition, the range bound must relate watermarked event-time +columns from both sides. The recommended, always-correct configuration -- watermarking both sides on +the event-time column used by the join, as in the example above -- satisfies both. Configurations +that leave the left state un-evicted (a watermark on only the right side of a range condition) or +leave either side unfiltered on the eviction key (a watermark on only one equality join key) are +rejected at analysis time. To restore the previous, looser behavior, set +`spark.sql.streaming.join.stricterWatermarkRequirements.enabled` to `false`; note that the looser +behavior can silently produce incorrect (missing) outer results. + ###### Semantic Guarantees of Stream-stream Outer Joins with Watermarking Outer joins have the same guarantees as [inner joins](#semantic-guarantees-of-stream-stream-inner-joins-with-watermarking) regarding watermark delays and whether data will be dropped or not. @@ -1318,6 +1332,15 @@ constraints must be specified for semi join. This is to evict unmatched input ro the engine must know when an input row on left side is not going to match with anything on right side in future. +As with a left outer join, the left state must be evicted so that never-matched left rows do not +accumulate: for an equality join the watermarked join key evicts the left key state; for a time +range condition the range bound must relate watermarked event-time columns from both sides. Unlike +outer and anti joins, a semi join has no additional late-filtering requirement, because it emits a +row on match rather than at eviction, so there is no already-emitted row for a late right row to +invalidate. A range condition watermarked only on one side is rejected at analysis time; to restore +the previous, looser behavior (which leaves the left state unbounded), set +`spark.sql.streaming.join.stricterWatermarkRequirements.enabled` to `false`. + ###### Semantic Guarantees of Stream-stream Semi Joins with Watermarking Semi joins have the same guarantees as [inner joins](#semantic-guarantees-of-stream-stream-inner-joins-with-watermarking) regarding watermark delays and whether data will be dropped or not. @@ -1331,19 +1354,18 @@ can arrive on the right side in future before it can emit the row. (The exceptio that fails a deterministic left-side-only predicate in the join condition: it can never match any right row, so it is emitted immediately without waiting for the watermark.) -Unlike outer joins, an anti join has stricter watermark requirements, because two independent things -must both hold. First, the right side must be late-filtered on the matching dimension, so that a -right row arriving too late to matter is dropped rather than processed; without this a late right -row could match a left row that has *already* been emitted as an anti row, silently corrupting the -result. Second, the left state must be evicted, since eviction is what emits the surviving unmatched -left rows. Concretely: for an equality join on a watermarked event-time key, that key must be -watermarked on the **right** side (the left state is then evicted through the shared key); for a -time range condition (for example `leftTime BETWEEN rightTime - INTERVAL 1 HOUR AND rightTime`), a -watermark must be defined on **both** sides -- the right side for late filtering and the left side -for eviction. The recommended, always-correct configuration is to watermark both sides on the -event-time column used by the join. Configurations that leave the right side un-filtered (for -example a watermark on only the left join key) or the left state never evicted (a watermark on only -the right side of a range condition) are rejected at analysis time. +Like a left outer join, an anti join has stricter watermark requirements, because two independent +things must both hold. First, both sides must be late-filtered on the matching dimension, so rows +that arrive too late to matter are dropped rather than processed; without this a late row could +match a left row that has *already* been emitted as an anti row, silently corrupting the result. +Second, the left state must be evicted, since eviction is what emits the surviving unmatched left +rows. Concretely: for an equality join on a watermarked event-time key, that key must be +watermarked on **both** sides; for a time range condition (for example `leftTime BETWEEN +rightTime - INTERVAL 1 HOUR AND rightTime`), the range bound must relate watermarked event-time +columns from both sides. The recommended, always-correct configuration is to watermark both sides on +the event-time column used by the join. Configurations that leave either side unfiltered on the +eviction key (for example a watermark on only one equality join key) or leave the left state never +evicted (a watermark on only one side of a range condition) are rejected at analysis time. Note that anti join is only supported in Append output mode. Update mode would have to emit rows early, before the watermark can rule out a future match, and such a row could be invalidated by a @@ -1439,8 +1461,9 @@ deterministic left-side-only predicate, which are emitted immediately as noted a @@ -1460,16 +1483,16 @@ deterministic left-side-only predicate, which are emitted immediately as noted a @@ -1484,7 +1507,8 @@ Additional details on supported joins: - Joins can be cascaded, that is, you can do `df1.join(df2, ...).join(df3, ...).join(df4, ....)`. -- As of Spark 2.4, you can use joins only when the query is in Append output mode. Other output modes are not yet supported. +- Inner and left semi joins support Append and Update output modes. Left outer, right outer, full + outer, and left anti joins support Append output mode only. Complete output mode is not supported. - You cannot use mapGroupsWithState and flatMapGroupsWithState before and after joins. @@ -2120,9 +2144,10 @@ Here is the compatibility matrix. - + diff --git a/docs/streaming/ss-migration-guide.md b/docs/streaming/ss-migration-guide.md index 210f752d9d17d..582eff56134b4 100644 --- a/docs/streaming/ss-migration-guide.md +++ b/docs/streaming/ss-migration-guide.md @@ -23,6 +23,10 @@ Note that this migration guide describes the items specific to Structured Stream Many items of SQL migration can be applied when migrating Structured Streaming to higher versions. Please refer [Migration Guide: SQL, Datasets and DataFrame](../sql-migration-guide.html). +## Upgrading from Structured Streaming 4.3 to 4.4 + +- Since Spark 4.4, stream-stream left semi and left outer joins enforce stricter watermark-placement requirements at analysis time, so that the left-side state their output (or bounded state size) depends on is actually evicted, and, for left outer, so that late rows cannot invalidate an already-emitted unmatched row. Configurations that previously ran but could silently produce incorrect results or unbounded state -- for example a range-condition join whose range bound is not between watermarked attributes on both sides, or a left outer equality join whose eviction key is not watermarked on both sides -- now fail with an `AnalysisException`. To restore the previous behavior for left semi and left outer joins, set `spark.sql.streaming.join.stricterWatermarkRequirements.enabled` to `false`. Newly supported stream-stream left anti joins always use the stricter requirements. (See [SPARK-58611](https://issues.apache.org/jira/browse/SPARK-58611) for more details.) + ## Upgrading from Structured Streaming 4.1 to 4.2 - Since Spark 4.2, restarting a streaming query from a checkpoint whose metadata file is missing while the offset or commit logs contain data fails with `STREAMING_CHECKPOINT_MISSING_METADATA_FILE`, instead of silently generating a new query ID (which can duplicate data in exactly-once sinks). Restore the metadata file or use a new checkpoint location. To restore the previous behavior, set `spark.sql.streaming.checkpoint.verifyMetadataExists.enabled` to `false`. (See [SPARK-55058](https://issues.apache.org/jira/browse/SPARK-55058) for more details.) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala index 929102745c143..597f4bac6a32a 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala @@ -34,6 +34,15 @@ import org.apache.spark.unsafe.types.CalendarInterval */ object StreamingJoinHelper extends PredicateHelper with Logging { + private def isWatermarked(expression: Expression): Boolean = expression match { + case ne: NamedExpression => ne.metadata.contains(EventTimeWatermark.delayKey) + case _ => false + } + + private def watermarkedAttributes(attributes: AttributeSet): AttributeSet = { + AttributeSet(attributes.filter(_.metadata.contains(delayKey)).toSeq) + } + /** * Check the provided logical plan to see if its join keys contain a watermark attribute. * @@ -52,37 +61,59 @@ object StreamingJoinHelper extends PredicateHelper with Logging { } /** - * Whether the watermark sits on the right equality join key at the ordinal that actually drives - * state-key eviction. This is stricter than [[isWatermarkInJoinKeys]] (side-insensitive) and than - * simply "some right key is watermarked" (ordinal-insensitive). It matters for join types (e.g. - * left anti) whose correctness depends on the right side being late-filtered on the *same* key - * dimension the left state is evicted by. + * Whether both equality join keys at the state-key eviction ordinal are watermarked. + * + * This is required by outer-like equality joins (left outer and left anti). Eviction of left + * state must be aligned with late-event filtering on both sides: a right watermark drops late + * right rows after unmatched rows have been emitted, and a left watermark drops late left rows + * after the right state that could have matched them has been evicted. * - * State-key eviction uses a single key ordinal, chosen exactly as in - * [[org.apache.spark.sql.execution.streaming.operators.stateful.join. - * StreamingSymmetricHashJoinHelper.findJoinKeyOrdinalForWatermark]]: the first watermarked left - * key, else the first watermarked right key. The right side is late-filtered on its own - * watermarked column, so unless the right key at that ordinal is the watermarked one, a right row - * that is old on the eviction key but fresh on some other watermarked key/column would not be - * late-filtered and could match an already-emitted anti row. This method must stay in sync with - * that ordinal selection. + * The eviction ordinal is chosen in the same way as + * StreamingSymmetricHashJoinHelper.findJoinKeyOrdinalForWatermark. */ - def isWatermarkOnRightEvictionJoinKey(plan: LogicalPlan): Boolean = { + def isWatermarkOnBothEvictionJoinKeys(plan: LogicalPlan): Boolean = { plan match { case ExtractEquiJoinKeys(_, leftKeys, rightKeys, _, _, _, _, _) => - def watermarked(e: Expression): Boolean = e match { - case ne: NamedExpression => ne.metadata.contains(EventTimeWatermark.delayKey) - case _ => false - } - val ordinal = leftKeys.indexWhere(watermarked) match { - case i if i >= 0 => i - case _ => rightKeys.indexWhere(watermarked) + joinKeyOrdinalForWatermark(leftKeys, rightKeys).exists { ordinal => + ordinal < leftKeys.length && ordinal < rightKeys.length && + isWatermarked(leftKeys(ordinal)) && isWatermarked(rightKeys(ordinal)) } - ordinal >= 0 && ordinal < rightKeys.length && watermarked(rightKeys(ordinal)) case _ => false } } + private def joinKeyOrdinalForWatermark( + leftKeys: Seq[Expression], + rightKeys: Seq[Expression]): Option[Int] = { + leftKeys.indexWhere(isWatermarked) match { + case i if i >= 0 => Some(i) + case _ => + rightKeys.indexWhere(isWatermarked) match { + case i if i >= 0 => Some(i) + case _ => None + } + } + } + + /** + * Like [[getStateValueWatermark]], but only succeeds when the state watermark is derived from + * watermarked attributes on both sides. This is useful for analysis-time validation of range + * conditions: the runtime value-watermark predicate is applied to the watermarked attribute on + * the side being evicted, so accepting a range bound over some other attribute would make the + * predicate either ineffective or incorrect. + */ + def getStateValueWatermarkOnWatermarkedAttributes( + attributesToFindStateWatermarkFor: AttributeSet, + attributesWithEventWatermark: AttributeSet, + joinCondition: Option[Expression], + eventWatermark: Option[Long]): Option[Long] = { + getStateValueWatermark( + watermarkedAttributes(attributesToFindStateWatermarkFor), + watermarkedAttributes(attributesWithEventWatermark), + joinCondition, + eventWatermark) + } + /** * Get state value watermark (see [[StreamingSymmetricHashJoinExec]] for context about it) * given the join condition and the event time watermark. This is how it works. diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala index 21891823c1e81..bd53b67b17d12 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala @@ -719,52 +719,113 @@ object UnsupportedOperationChecker extends Logging { "the nullable side and an appropriate range condition")(join) } - // Left anti has stricter watermark requirements than the other stream-stream joins, because it - // is the only outer-like join whose output (the unmatched left rows) is produced from left - // state eviction AND can be invalidated by a *later* right row. Correct anti output needs both - // of the following, which the generic side-insensitive check above does not guarantee: + // The generic check above is side- and ordinal-insensitive: it accepts any watermark in the + // join keys, or any watermark that yields a state value watermark on the nullable side. That is + // not enough for the join types whose output (or bounded state size) depends on the left state + // being evicted: left semi, left outer, and left anti. // - // 1. The right side must be late-filtered on the matching dimension, so a right row old - // enough to match an already-evicted (hence already-emitted) left row is dropped rather - // than processed. The operator late-filters a side using that side's own watermarked - // event-time column, so this bounds matching only when the watermark is on the right - // *join key* (equi-join path) or the right *range-bound* attribute (range path). - // 2. The left state must actually be evicted, so surviving unmatched left rows are emitted. - // A watermark in the join keys evicts the left key state; a range condition needs a - // watermark on the LEFT side to evict the left value state. - // - // Left outer/semi keep their existing one-sided-watermark behavior for compatibility; a general - // fix for them is tracked separately. Note the range path still only checks that the left side - // is watermarked, not that the watermark is on the range-bound attribute -- that column-level - // precision is a pre-existing gap shared by all stream-stream joins and is deferred. - if (join.joinType == LeftAnti) { + // Left semi/outer are existing join types, so users can temporarily restore the previous loose + // behavior with SQLConf.STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED. Left anti is + // new, so it always uses the stricter rules. + if (join.joinType == LeftAnti || SQLConf.get.streamingJoinStricterWatermarkRequirements) { + checkStreamStreamJoinWatermarkPlacement( + join, + watermarkInJoinKeys, + canRestorePreviousBehavior = join.joinType != LeftAnti) + } + } + + /** + * Enforce stricter watermark-placement requirements for the stream-stream join types whose + * correctness or bounded state depends on the left-side state being evicted: left semi, left + * outer, and left anti. These requirements are stricter than the generic check in + * [[checkForStreamStreamJoinWatermark]]. For left semi and left outer, these checks are gated + * by [[SQLConf.STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED]]. Left anti is new and + * always enforces these requirements. + * + * Two independent requirements apply: + * + * 1. Left-state eviction. Left semi/outer/anti all rely on the left state being evicted: left + * outer/anti emit their surviving unmatched left rows from that eviction (so without it the + * unmatched/outer output is never produced), and left semi cleans up its never-matched left + * rows there (so without it the left state grows without bound). In the equi-join path the + * left key state is evicted through the shared watermarked join key. In the range-condition + * path the left *value* state is evicted only when the range condition derives the left state + * watermark from watermarked attributes on both sides. + * + * 2. No invalidation by late rows. This applies only to the outer-like types (left outer and + * left anti), whose already-emitted unmatched row can be invalidated by a later match. Both + * sides must be late-filtered on the same dimension the left state is evicted by; otherwise + * a row that is old on the eviction key can still match an already-emitted unmatched row. In + * the equi-join path this requires both join keys at the eviction ordinal to be watermarked. + * Left semi does not need this: it emits on match while joining, never at eviction, so there + * is nothing to invalidate. + */ + private def checkStreamStreamJoinWatermarkPlacement( + join: Join, + watermarkInJoinKeys: Boolean, + canRestorePreviousBehavior: Boolean): Unit = { + // (requiresLeftStateEviction, isOuterLike) per join type. Only the left-side-eviction join + // types are constrained here; inner, right outer, and full outer keep their existing behavior. + val (requiresLeftStateEviction, isOuterLike) = join.joinType match { + case LeftOuter => (true, true) + case LeftAnti => (true, true) + case LeftSemi => (true, false) + case _ => (false, false) + } + + if (requiresLeftStateEviction) { if (watermarkInJoinKeys) { - // Equi-join path: the left key state is evicted via the join-key watermark at a single - // ordinal, so we need the right side late-filtered on that *same* key ordinal. That - // requires the right key at the eviction ordinal to be watermarked -- a watermark on only - // the left key, on an unrelated right column, or on a different key position (composite - // keys) would let a late right row match an already-emitted anti row. - if (!StreamingJoinHelper.isWatermarkOnRightEvictionJoinKey(join)) { + // Equi-join path: the left key state is evicted via the join-key watermark, so the + // eviction requirement is met. For the outer-like types we additionally need both sides + // late-filtered on the same key ordinal used for eviction. A watermark on only one side, on + // an unrelated column, or on a different key position (composite keys) would let a late row + // match an already-emitted unmatched row. + if (isOuterLike && !StreamingJoinHelper.isWatermarkOnBothEvictionJoinKeys(join)) { + val restore = if (canRestorePreviousBehavior) { + " To restore the previous behavior, set " + + s"${SQLConf.STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED.key}=false." + } else { + "" + } throwError( - "Stream-stream LeftAnti join between two streaming DataFrame/Datasets with a " + - "watermarked join key requires the watermark to be on the right join key used for " + - "state eviction (for a composite key, the right key at the same position as the " + - "watermarked left key), so that late right rows are dropped and cannot invalidate " + - "already-emitted anti rows. A watermark on only the left key, on an unrelated " + - "right column, or on a different key position, is not supported.")(join) + s"Stream-stream ${join.joinType} join between two streaming DataFrame/Datasets with " + + "a " + + "watermarked join key requires watermarks on both join keys used for state " + + "eviction (for a composite key, the left and right keys at the same position), so " + + "that late rows on either side are dropped and cannot invalidate already-emitted " + + "unmatched " + + s"rows. A watermark on only one side is not supported.$restore")(join) } } else { - // Range-condition path (hasValidWatermarkRange holds here): the right side is late-filtered - // via its range-bound watermark, but the left state is evicted -- and hence anti rows - // emitted -- only when the LEFT side is watermarked. - val leftHasWatermark = - join.left.output.exists(_.metadata.contains(EventTimeWatermark.delayKey)) - if (!leftHasWatermark) { + // Range-condition path (hasValidWatermarkRange holds here): the left state is evicted only + // when the range condition derives its state watermark from watermarked attributes on both + // sides. A right-only watermark, or a watermark on an unrelated left/right attribute, + // leaves the runtime eviction predicate ineffective or incorrect. + val hasWatermarkRangeOnBothSides = + StreamingJoinHelper.getStateValueWatermarkOnWatermarkedAttributes( + join.left.outputSet, join.right.outputSet, join.condition, Some(1000000)).isDefined + if (!hasWatermarkRangeOnBothSides) { + val reason = + if (isOuterLike) { + "so that the left state whose surviving unmatched rows form the " + + s"${join.joinType} output is evicted" + } else { + "so that the left state is evicted and does not grow without bound" + } + val restore = if (canRestorePreviousBehavior) { + " To restore the previous behavior, set " + + s"${SQLConf.STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED.key}=false." + } else { + "" + } throwError( - "Stream-stream LeftAnti join between two streaming DataFrame/Datasets with a range " + - "condition requires a watermark on the left side, so that the left state (whose " + - "surviving unmatched rows form the anti output) is evicted. A watermark on only " + - "the right side is not supported.")(join) + s"Stream-stream ${join.joinType} join between two streaming DataFrame/Datasets with " + + "a " + + "range condition requires a range bound between watermarked attributes on both " + + s"sides, $reason. A watermark on only one side, or on an unrelated attribute, is " + + "not " + + s"supported.$restore")(join) } } } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 8a0d88346b63c..52576a0d62da2 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -3784,6 +3784,21 @@ object SQLConf { .checkValue(v => Set(1, 2, 3, 4).contains(v), "Valid versions are 1, 2, 3, and 4") .createWithDefault(2) + val STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED = + buildConf("spark.sql.streaming.join.stricterWatermarkRequirements.enabled") + .doc("When true, the analyzer enforces stricter watermark-placement requirements for " + + "stream-stream left semi and left outer joins, so that the left-side state which their " + + "output (or bounded state size) depends on is actually evicted, and, for left outer, so " + + "that late rows cannot invalidate an already-emitted unmatched row. Configurations that " + + "would otherwise silently produce incorrect results or unbounded state (e.g. a " + + "range-condition join whose range bound is not between watermarked attributes on both " + + "sides, or a left outer equality join whose eviction key is not watermarked on both " + + "sides) are rejected. Set this to false to restore the previous, looser behavior for " + + "left semi and left outer joins. Left anti joins always use the stricter requirements.") + .version("4.4.0") + .booleanConf + .createWithDefault(true) + val STREAMING_SESSION_WINDOW_MERGE_SESSIONS_IN_LOCAL_PARTITION = buildConf("spark.sql.streaming.sessionWindow.merge.sessions.in.local.partition") .doc("When true, streaming session window sorts and merge sessions in local partition " + @@ -8500,6 +8515,9 @@ class SQLConf extends Serializable with Logging with SqlApiConf { def stateStoreSkipNullsForStreamStreamJoins: Boolean = getConf(STATE_STORE_SKIP_NULLS_FOR_STREAM_STREAM_JOINS) + def streamingJoinStricterWatermarkRequirements: Boolean = + getConf(STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED) + def stateStoreCoordinatorMultiplierForMinVersionDiffToLog: Long = getConf(STATE_STORE_COORDINATOR_MULTIPLIER_FOR_MIN_VERSION_DIFF_TO_LOG) diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala index c1c549f5a03f2..39b0537f1de87 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala @@ -481,10 +481,11 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { Seq("is not supported in Complete output mode", allowedModesMsg)) } - // Left outer, right outer, full outer, left semi joins. - // Left anti is handled separately below: it has stricter watermark requirements (see - // checkForStreamStreamJoinWatermark), so the side-insensitive cases here do not all apply to it. - Seq(LeftOuter, RightOuter, FullOuter, LeftSemi).foreach { joinType => + // Right outer and full outer joins keep the side-insensitive watermark behavior. Left outer, + // left semi, and left anti are handled in dedicated blocks below: they have stricter + // watermark-placement requirements (see checkStreamStreamJoinWatermarkPlacement), so the + // side-insensitive cases here do not all apply to them. + Seq(RightOuter, FullOuter).foreach { joinType => // Stream-stream allowed with join on watermark attribute // Note that the attribute need not be watermarked on both sides. assertSupportedInStreamingPlan( @@ -531,25 +532,173 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { "the nullable side and an appropriate range condition")) } - // Left anti join: stricter watermark requirements than the other join types. Correct anti output - // needs (1) the right side late-filtered on the matching dimension -- the right join key on the - // equi-join path, or the right range-bound attribute on the range path -- so a late right row - // cannot invalidate an already-emitted anti row, and (2) left-state eviction, via a join-key - // watermark or a watermark on the left side. Distinct attributes are used per side to avoid - // exprId collisions in these synthetic plans. + // Left outer and left semi joins: stricter watermark-placement requirements, shared with left + // anti (see checkStreamStreamJoinWatermarkPlacement) and gated by + // STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED. Their left-side state must be + // evicted -- left outer emits its unmatched rows from that eviction, left semi cleans up its + // never-matched rows there -- and left outer, being outer-like, additionally needs both sides + // late-filtered on the eviction key so late rows cannot invalidate already-emitted unmatched + // rows. Distinct attributes per side avoid exprId collisions in these synthetic plans. + { + val leftWm = AttributeReference("olw", IntegerType)().withMetadata(watermarkMetadata) + val rightWm = AttributeReference("orw", IntegerType)().withMetadata(watermarkMetadata) + val leftPlain = AttributeReference("olp", IntegerType)() + val rightPlain = AttributeReference("orp", IntegerType)() + val leftOtherWm = AttributeReference("olo", IntegerType)().withMetadata(watermarkMetadata) + val rightOtherWm = AttributeReference("oro", IntegerType)().withMetadata(watermarkMetadata) + val strictKey = SQLConf.STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED.key + + Seq(LeftOuter, LeftSemi).foreach { joinType => + // Supported: range condition with a watermark on both sides (the left value state is evicted + // on its own watermark, derived from the right watermark). + assertSupportedInStreamingPlan( + s"$joinType join with stream-stream relations and range condition, watermark on both sides", + new TestStreamingRelation(leftWm).join(new TestStreamingRelation(rightWm), + joinType = joinType, condition = Some(leftWm > rightWm + 10)), + OutputMode.Append()) + + // Not supported: range condition with a watermark on the right only -- the left state is + // never evicted (so left outer never emits its unmatched rows, and left semi's left state + // grows without bound). + assertNotSupportedInStreamingPlan( + s"$joinType join with stream-stream relations and range condition, right watermark only", + new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightWm), + joinType = joinType, condition = Some(leftPlain > rightWm + 10)), + OutputMode.Append(), + Seq("requires a range bound between watermarked attributes on both sides")) + + // Not supported: the left watermark is unrelated to the range-bound left attribute, so the + // runtime eviction predicate would be applied to the wrong column. + assertNotSupportedInStreamingPlan( + s"$joinType join with stream-stream relations and range condition, unrelated left " + + "watermark", + new TestStreamingRelation(Seq(leftPlain, leftOtherWm)).join( + new TestStreamingRelation(rightWm), + joinType = joinType, condition = Some(leftPlain > rightWm + 10)), + OutputMode.Append(), + Seq("requires a range bound between watermarked attributes on both sides")) + + // Not supported: the right watermark is unrelated to the range-bound right attribute, so the + // state watermark cannot be derived from the runtime late-filtering column. + assertNotSupportedInStreamingPlan( + s"$joinType join with stream-stream relations and range condition, unrelated right " + + "watermark", + new TestStreamingRelation(leftWm).join( + new TestStreamingRelation(Seq(rightPlain, rightOtherWm)), + joinType = joinType, condition = Some(leftWm > rightPlain + 10)), + OutputMode.Append(), + Seq("without a watermark in the join keys")) + + // Kill switch: with the stricter requirements disabled, the right-only range config falls + // back to the previous (looser) behavior and is accepted again. + assertSupportedInStreamingPlan( + s"$joinType join, right watermark only, accepted when stricter requirements disabled", + new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightWm), + joinType = joinType, condition = Some(leftPlain > rightWm + 10)), + OutputMode.Append(), + strictKey -> "false") + } + + // Left semi is not outer-like: it emits on match, never at eviction, so the equi-join path has + // no no-invalidation requirement. A one-sided key watermark is enough to evict the left key + // state through the shared equality key. + assertSupportedInStreamingPlan( + "left semi join with stream-stream relations and join key watermarked on the right side only", + new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightWm), + joinType = LeftSemi, condition = Some(leftPlain === rightWm)), + OutputMode.Append()) + assertSupportedInStreamingPlan( + "left semi join with stream-stream relations and join key watermarked on the left side only", + new TestStreamingRelation(leftWm).join(new TestStreamingRelation(rightPlain), + joinType = LeftSemi, condition = Some(leftWm === rightPlain)), + OutputMode.Append()) + + // Left outer is outer-like, so equality joins require both keys at the eviction ordinal to be + // watermarked. Otherwise one side is not late-filtered on the dimension used to evict state. + assertSupportedInStreamingPlan( + "left outer join with stream-stream relations and both join keys watermarked", + new TestStreamingRelation(leftWm).join(new TestStreamingRelation(rightWm), + joinType = LeftOuter, condition = Some(leftWm === rightWm)), + OutputMode.Append()) + Seq( + ("right side only", leftPlain, rightWm), + ("left side only", leftWm, rightPlain) + ).foreach { case (side, leftKey, rightKey) => + assertNotSupportedInStreamingPlan( + s"left outer join with stream-stream relations and join key watermarked on $side", + new TestStreamingRelation(leftKey).join(new TestStreamingRelation(rightKey), + joinType = LeftOuter, condition = Some(leftKey === rightKey)), + OutputMode.Append(), + Seq("requires watermarks on both join keys used for state eviction")) + } + + assertNotSupportedInStreamingPlan( + "left outer join with stream-stream relations and watermark on an unrelated right column", + new TestStreamingRelation(leftWm).join( + new TestStreamingRelation(Seq(rightPlain, rightWm)), + joinType = LeftOuter, condition = Some(leftWm === rightPlain)), + OutputMode.Append(), + Seq("requires watermarks on both join keys used for state eviction")) + + // Composite equality keys. State-key eviction uses a single ordinal (the first watermarked + // left key), so both keys at that ordinal must be watermarked. + val outerLeftWm1 = AttributeReference("olw1", IntegerType)().withMetadata(watermarkMetadata) + val outerLeftPlain2 = AttributeReference("olp2", IntegerType)() + val outerRightWm1 = AttributeReference("orw1", IntegerType)().withMetadata(watermarkMetadata) + val outerRightPlain1 = AttributeReference("orp1", IntegerType)() + val outerRightPlain2 = AttributeReference("orp2", IntegerType)() + val outerRightWm2 = AttributeReference("orw2", IntegerType)().withMetadata(watermarkMetadata) + + assertSupportedInStreamingPlan( + "left outer join with stream-stream relations and composite key watermarked at same ordinal", + new TestStreamingRelation(Seq(outerLeftWm1, outerLeftPlain2)).join( + new TestStreamingRelation(Seq(outerRightWm1, outerRightPlain2)), + joinType = LeftOuter, + condition = Some(outerLeftWm1 === outerRightWm1 && + outerLeftPlain2 === outerRightPlain2)), + OutputMode.Append()) + + assertNotSupportedInStreamingPlan( + "left outer join with stream-stream relations and composite key watermarked on mismatched " + + "ordinals", + new TestStreamingRelation(Seq(outerLeftWm1, outerLeftPlain2)).join( + new TestStreamingRelation(Seq(outerRightPlain1, outerRightWm2)), + joinType = LeftOuter, + condition = Some(outerLeftWm1 === outerRightPlain1 && + outerLeftPlain2 === outerRightWm2)), + OutputMode.Append(), + Seq("requires watermarks on both join keys used for state eviction")) + + // Kill switch: accepted again when the stricter requirements are disabled. + assertSupportedInStreamingPlan( + "left outer join, right-only key watermark, accepted when stricter requirements disabled", + new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightWm), + joinType = LeftOuter, condition = Some(leftPlain === rightWm)), + OutputMode.Append(), + strictKey -> "false") + assertSupportedInStreamingPlan( + "left outer join, left-only key watermark, accepted when stricter requirements disabled", + new TestStreamingRelation(leftWm).join(new TestStreamingRelation(rightPlain), + joinType = LeftOuter, condition = Some(leftWm === rightPlain)), + OutputMode.Append(), + strictKey -> "false") + } + + // Left anti join uses the same strict requirements as left outer, but it is new in this PR, so + // the compatibility flag does not disable them. { val leftWm = AttributeReference("lw", IntegerType)().withMetadata(watermarkMetadata) val rightWm = AttributeReference("rw", IntegerType)().withMetadata(watermarkMetadata) val leftPlain = AttributeReference("lp", IntegerType)() val rightPlain = AttributeReference("rp", IntegerType)() + val leftOtherWm = AttributeReference("lo", IntegerType)().withMetadata(watermarkMetadata) val rightOtherWm = AttributeReference("ro", IntegerType)().withMetadata(watermarkMetadata) - // Supported: equality join key watermarked on the right (the left key state is evicted via the - // shared key watermark, and the right side is late-filtered on the join key). + // Supported: equality join keys watermarked on both sides. assertSupportedInStreamingPlan( - "left anti join with stream-stream relations and join key watermarked on the right side", - new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightWm), - joinType = LeftAnti, condition = Some(leftPlain === rightWm)), + "left anti join with stream-stream relations and both join keys watermarked", + new TestStreamingRelation(leftWm).join(new TestStreamingRelation(rightWm), + joinType = LeftAnti, condition = Some(leftWm === rightWm)), OutputMode.Append()) // Supported: range condition with a watermark on both sides. @@ -559,14 +708,17 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { joinType = LeftAnti, condition = Some(leftWm > rightWm + 10)), OutputMode.Append()) - // Not supported: equality join key watermarked on the left only -- the right join key is not - // late-filtered, so a late right row could invalidate an already-emitted anti row. - assertNotSupportedInStreamingPlan( - "left anti join with stream-stream relations and join key watermarked on the left side only", - new TestStreamingRelation(leftWm).join(new TestStreamingRelation(rightPlain), - joinType = LeftAnti, condition = Some(leftWm === rightPlain)), - OutputMode.Append(), - Seq("requires the watermark to be on the right join key")) + Seq( + ("right side only", leftPlain, rightWm), + ("left side only", leftWm, rightPlain) + ).foreach { case (side, leftKey, rightKey) => + assertNotSupportedInStreamingPlan( + s"left anti join with stream-stream relations and join key watermarked on $side", + new TestStreamingRelation(leftKey).join(new TestStreamingRelation(rightKey), + joinType = LeftAnti, condition = Some(leftKey === rightKey)), + OutputMode.Append(), + Seq("requires watermarks on both join keys used for state eviction")) + } // Not supported: the watermark is on an unrelated right column, not the right join key, so the // right side is not late-filtered on the key that bounds matching. @@ -576,10 +728,10 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { new TestStreamingRelation(Seq(rightPlain, rightOtherWm)), joinType = LeftAnti, condition = Some(leftWm === rightPlain)), OutputMode.Append(), - Seq("requires the watermark to be on the right join key")) + Seq("requires watermarks on both join keys used for state eviction")) // Composite equality keys. State-key eviction uses a single ordinal (the first watermarked - // left key), so the right key at that ordinal must be the watermarked one. + // left key), so both keys at that ordinal must be watermarked. val leftWm1 = AttributeReference("lw1", IntegerType)().withMetadata(watermarkMetadata) val leftPlain2 = AttributeReference("lp2", IntegerType)() val rightWm1 = AttributeReference("rw1", IntegerType)().withMetadata(watermarkMetadata) @@ -607,7 +759,7 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { joinType = LeftAnti, condition = Some(leftWm1 === rightPlain1 && leftPlain2 === rightWm2)), OutputMode.Append(), - Seq("requires the watermark to be on the right join key")) + Seq("requires watermarks on both join keys used for state eviction")) // Not supported: range condition with a watermark on the right only -- the left state is never // evicted, so no anti row is ever produced. @@ -616,7 +768,53 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { new TestStreamingRelation(leftPlain).join(new TestStreamingRelation(rightWm), joinType = LeftAnti, condition = Some(leftPlain > rightWm + 10)), OutputMode.Append(), - Seq("requires a watermark on the left side")) + Seq("requires a range bound between watermarked attributes on both sides")) + + // Not supported: the left watermark is unrelated to the range-bound left attribute. + assertNotSupportedInStreamingPlan( + "left anti join with stream-stream relations and range condition, unrelated left watermark", + new TestStreamingRelation(Seq(leftPlain, leftOtherWm)).join( + new TestStreamingRelation(rightWm), + joinType = LeftAnti, condition = Some(leftPlain > rightWm + 10)), + OutputMode.Append(), + Seq("requires a range bound between watermarked attributes on both sides")) + + // Not supported: the right watermark is unrelated to the range-bound right attribute. + assertNotSupportedInStreamingPlan( + "left anti join with stream-stream relations and range condition, unrelated right watermark", + new TestStreamingRelation(leftWm).join( + new TestStreamingRelation(Seq(rightPlain, rightOtherWm)), + joinType = LeftAnti, condition = Some(leftWm > rightPlain + 10)), + OutputMode.Append(), + Seq("without a watermark in the join keys")) + + test("streaming plan - left anti strict watermark rules ignore compatibility flag") { + withSQLConf(SQLConf.STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED.key -> "false") { + val rangeError = intercept[AnalysisException] { + UnsupportedOperationChecker.checkForStreaming( + wrapInStreaming(new TestStreamingRelation(leftPlain).join( + new TestStreamingRelation(rightWm), + joinType = LeftAnti, + condition = Some(leftPlain > rightWm + 10))), + OutputMode.Append()) + } + assert(rangeError.getMessage.contains( + "requires a range bound between watermarked attributes on both sides")) + + Seq((leftPlain, rightWm), (leftWm, rightPlain)).foreach { case (leftKey, rightKey) => + val equalityError = intercept[AnalysisException] { + UnsupportedOperationChecker.checkForStreaming( + wrapInStreaming(new TestStreamingRelation(leftKey).join( + new TestStreamingRelation(rightKey), + joinType = LeftAnti, + condition = Some(leftKey === rightKey))), + OutputMode.Append()) + } + assert(equalityError.getMessage.contains( + "requires watermarks on both join keys used for state eviction")) + } + } + } // Not supported: no watermark at all (rejected by the generic stream-stream join check). assertNotSupportedInStreamingPlan( diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala index 083789b10e1cf..62af3833fc45f 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala @@ -885,7 +885,8 @@ case class StreamingSymmetricHashJoinExec( otherSideJoiner.joinStateManager.getJoinedRowsAndRemoveMatched( key, thatRow => generateJoinedRow(thisRow, thatRow), - postJoinFilter).map { row => + postJoinFilter, + timestampRange = computeTimestampRange(thisRow)).map { row => numRemovedFromOtherSideDuringJoinCount += 1 row } @@ -947,13 +948,15 @@ case class StreamingSymmetricHashJoinExec( // and the join type is left semi. // For other cases, the input should be added, including the case it's going to be evicted // in this batch. It hasn't yet evaluated with inputs from right side for this batch. - // Refer to the classdoc of SteramingSymmetricHashJoinExec about how stream-stream join + // Refer to the classdoc of StreamingSymmetricHashJoinExec about how stream-stream join // works. - // - Right side: for this side, the evaluation with inputs from left side for this batch - // is done at this point. That said, input can be skipped to be added to the state store - // if input is going to be evicted in this batch. Though, input should be added to the - // state store if it's right outer join or full outer join, as unmatched output is - // handled during state eviction. + // - Right side: for this side, the evaluation with earlier inputs from left side for this + // batch is done at this point. That said, input can be skipped to be added to the state + // store if input is going to be evicted in this batch. Two join types still need to store + // such rows temporarily: + // - right/full outer, because unmatched output is handled during state eviction; + // - left semi/anti, because the right side is processed first and later left input from + // the same batch must still be able to match these non-late right rows. // // Left anti gets the same left-side skip-on-match treatment as left semi: a matched left // row can never be anti output, so there is no reason to keep it in state. @@ -969,8 +972,11 @@ case class StreamingSymmetricHashJoinExec( // if the input is not evicted in this batch (hence need to be persisted) val isNotEvictingInThisBatch = !stateKeyWatermarkPredicateFunc(key) && !stateValueWatermarkPredicateFunc(thisRow) + val isRightSideOfLeftSemiOrAnti = + joinType == LeftSemi || joinType == LeftAnti isNotEvictingInThisBatch || + isRightSideOfLeftSemiOrAnti || // if the input is producing "unmatched row" in this batch ( (joinType == RightOuter && !iteratorNotEmpty) || diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala index 97878b33c778f..46ef6c3835bab 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala @@ -94,11 +94,14 @@ trait SymmetricHashJoinStateManager { * * It is caller's responsibility to consume the whole iterator. * - * NOTE: For the rows which already have been marked as matched in the state, this method removes - * them from the state without returning them. Under normal operation, this should not happen and - * this should not be an issue, but it can occur if the join type was changed during query - * restart. We do not define an expected behavior for such changes, so we just optimize it rather - * than trying to provide some best-effort results. + * @param timestampRange optional inclusive timestamp range for implementations that can prune the + * state scan by event time. + * + * NOTE: For the rows which already have been marked as matched in the scanned state, this method + * removes them from the state without returning them. Under normal operation, this should not + * happen and this should not be an issue, but it can occur if the join type was changed during + * query restart. We do not define an expected behavior for such changes, so we just optimize it + * rather than trying to provide some best-effort results. * * NOTE2: There is a further optimization opportunity -- if a row does not pass the predicate * (postJoinFilter), it may never match in future batches as long as expressions are @@ -110,7 +113,8 @@ trait SymmetricHashJoinStateManager { def getJoinedRowsAndRemoveMatched( key: UnsafeRow, generateJoinedRow: InternalRow => JoinedRow, - predicate: JoinedRow => Boolean): Iterator[JoinedRow] + predicate: JoinedRow => Boolean, + timestampRange: Option[(Long, Long)] = None): Iterator[JoinedRow] /** * Provide all key-value pairs in the state manager. @@ -518,7 +522,8 @@ class SymmetricHashJoinStateManagerV4( override def getJoinedRowsAndRemoveMatched( key: UnsafeRow, generateJoinedRow: InternalRow => JoinedRow, - predicate: JoinedRow => Boolean): Iterator[JoinedRow] = { + predicate: JoinedRow => Boolean, + timestampRange: Option[(Long, Long)]): Iterator[JoinedRow] = { def getJoinedRowsFromTsAndValues( ts: Long, valuesAndMatched: Array[ValueAndMatchPair]): Iterator[JoinedRow] = { @@ -578,7 +583,8 @@ class SymmetricHashJoinStateManagerV4( getJoinedRowsFromTsAndValues(ts, valuesAndMatchedIter.toArray) case _ => - keyWithTsToValues.getValues(key).flatMap { result => + val (minTs, maxTs) = timestampRange.getOrElse((Long.MinValue, Long.MaxValue)) + keyWithTsToValues.getValuesInRange(key, minTs, maxTs).flatMap { result => val ts = result.timestamp val valuesAndMatched = result.values.toArray getJoinedRowsFromTsAndValues(ts, valuesAndMatched) @@ -1191,7 +1197,8 @@ abstract class SymmetricHashJoinStateManagerBase( override def getJoinedRowsAndRemoveMatched( key: UnsafeRow, generateJoinedRow: InternalRow => JoinedRow, - predicate: JoinedRow => Boolean): Iterator[JoinedRow] = { + predicate: JoinedRow => Boolean, + timestampRange: Option[(Long, Long)]): Iterator[JoinedRow] = { new NextIterator[JoinedRow] { private var numValues: Long = keyToNumValues.get(key) private var index: Long = 0L diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala index 0460a41f4cc5b..acc32c89908ff 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala @@ -1063,6 +1063,20 @@ class SymmetricHashJoinStateManagerEventTimeInValueSuite ).map(_.getInt(1)).toSeq.sorted } + private def removeMatchedRowTimestamps( + key: Int, + range: Option[(Long, Long)], + predicate: JoinedRow => Boolean = (_: JoinedRow) => true)( + implicit manager: SymmetricHashJoinStateManager): Seq[Int] = { + val dummyRow = new GenericInternalRow(0) + manager.getJoinedRowsAndRemoveMatched( + toJoinKeyRow(key), + row => new JoinedRow(row, dummyRow), + predicate, + timestampRange = range + ).map(_.getInt(1)).toSeq.sorted + } + test("StreamingJoinStateManager V4 - getJoinedRows with timestampRange") { withJoinStateManager( inputValueAttributes, joinKeyExpressions, stateFormatVersion = 4) { manager => @@ -1082,6 +1096,27 @@ class SymmetricHashJoinStateManagerEventTimeInValueSuite } } + test("StreamingJoinStateManager V4 - getJoinedRowsAndRemoveMatched with timestampRange") { + withJoinStateManager( + inputValueAttributes, joinKeyExpressions, stateFormatVersion = 4) { manager => + implicit val mgr = manager + + manager.append(toJoinKeyRow(40), toInputValue(40, 5), matched = true) + Seq(10, 20, 30, 40, 50).foreach(append(40, _)) + + assert(removeMatchedRowTimestamps(40, Some((20L, 40L))) === Seq(20, 30, 40)) + assert(get(40) === Seq(5, 10, 50)) + + Seq(20, 30, 40).foreach(append(40, _)) + val matched = removeMatchedRowTimestamps( + 40, + Some((20L, 40L)), + joinedRow => joinedRow.getInt(1) != 30) + assert(matched === Seq(20, 40)) + assert(get(40) === Seq(5, 10, 30, 50)) + } + } + test("StreamingJoinStateManager V4 - timestampRange with multiple values per timestamp") { withJoinStateManager( inputValueAttributes, joinKeyExpressions, stateFormatVersion = 4) { manager => diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala index 636270a0df8ff..708a7b85e26ed 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala @@ -249,9 +249,8 @@ abstract class StreamingJoinSuite val df1Base = leftInput.toDF().toDF("leftKey", "time") .select($"leftKey", timestamp_seconds($"time") as "leftTime", ($"leftKey" * 2) as "leftValue") - // The left watermark is optional for most join types; left anti requires it (a watermark on - // only the right side leaves the left state unevicted). Allow tests to omit it to exercise - // that path. + // Left semi/outer/anti require range bounds over watermarked attributes on both sides. Allow + // tests to omit the left watermark to exercise that analyzer path. val df1 = if (leftWatermark) df1Base.withWatermark("leftTime", watermark) else df1Base val df2 = rightInput.toDF().toDF("rightKey", "time") @@ -276,6 +275,23 @@ abstract class StreamingJoinSuite (leftInput, rightInput, select) } + protected def setupEqualityJoinWithZeroWatermark(joinType: String) + : (MemoryStream[(String, Int)], MemoryStream[(String, Int)], DataFrame) = { + val leftInput = MemoryStream[(String, Int)] + val rightInput = MemoryStream[(String, Int)] + + val left = leftInput.toDF() + .selectExpr("_1 AS key", "timestamp_seconds(_2) AS eventTime") + .withWatermark("eventTime", "0 seconds") + val right = rightInput.toDF() + .selectExpr("_1 AS key", "timestamp_seconds(_2) AS eventTime") + .withWatermark("eventTime", "0 seconds") + val joined = left.join(right, Seq("key", "eventTime"), joinType) + .selectExpr("key", "CAST(eventTime AS long) AS eventTime") + + (leftInput, rightInput, joined) + } + protected def setupSelfJoin(joinType: String) : (MemoryStream[(Int, Long)], DataFrame) = { @@ -1987,6 +2003,57 @@ abstract class StreamingOuterJoinSuite extends StreamingOuterJoinBase { } } } + + test("left outer join with a range condition requires watermarked bounds on both sides") { + // Only the right side is watermarked. With a range condition (no watermark in the join keys), + // the operator never builds a left-side eviction predicate, so the unmatched (null-extended) + // left rows would never be emitted. This must be rejected at analysis time rather than silently + // dropping the outer output. + val (_, _, joined) = setupJoinWithRangeCondition("left_outer", leftWatermark = false) + + val e = intercept[AnalysisException] { + joined.writeStream.format("memory").queryName("leftOuterRightWatermarkOnly") + .outputMode(OutputMode.Append()).start() + } + assert(e.getMessage.contains( + "requires a range bound between watermarked attributes on both sides")) + } + + test("left outer join with an equi join key watermarked on the left only is rejected") { + // The join key is watermarked on the left only, so the right side is not late-filtered on the + // eviction key: a late right row could match a left row already emitted as a null-extended + // outer row. Rejected at analysis time. + val leftInput = MemoryStream[(Int, Int)] + val rightInput = MemoryStream[(Int, Int)] + val df1 = leftInput.toDF().toDF("key", "time") + .select($"key", timestamp_seconds($"time") as "leftTime") + .withWatermark("leftTime", "10 seconds") + val df2 = rightInput.toDF().toDF("key", "time") + .select($"key", timestamp_seconds($"time") as "rightTime") + val joined = df1.join(df2, expr("leftTime = rightTime"), "left_outer") + + val e = intercept[AnalysisException] { + joined.writeStream.format("memory").queryName("leftOuterLeftKeyWatermarkOnly") + .outputMode(OutputMode.Append()).start() + } + assert(e.getMessage.contains("requires watermarks on both join keys used for state eviction")) + } + + test("stricter watermark requirements can be disabled for left outer join") { + // With the kill switch off, the right-only range config that is otherwise rejected is accepted + // again (falling back to the previous, looser behavior). + withSQLConf( + SQLConf.STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED.key -> "false") { + val (_, _, joined) = setupJoinWithRangeCondition("left_outer", leftWatermark = false) + val query = joined.writeStream.format("memory") + .queryName("leftOuterFallback").outputMode(OutputMode.Append()).start() + try { + query.processAllAvailable() + } finally { + query.stop() + } + } + } } @SlowSQLTest @@ -2430,6 +2497,36 @@ abstract class StreamingLeftSemiJoinBase extends StreamingJoinSuite { ) } + testWithAppendAndUpdate("left semi join matches same-batch right row between late and " + + "eviction watermarks") { outputMode => + withTempDir { checkpoint => + withSQLConf(SQLConf.STREAMING_NO_DATA_MICRO_BATCHES_ENABLED.key -> "false") { + val (leftInput, rightInput, joined) = + setupEqualityJoinWithZeroWatermark("left_semi") + + testStream(joined, outputMode)( + StartStream(checkpointLocation = checkpoint.getCanonicalPath), + // batch 0 + // WM: late record = 0, eviction = 0 + MultiAddData( + (leftInput, Seq(("a", 1), ("b", 2))), + (rightInput, Seq(("b", 2), ("c", 1))) + ), + CheckNewAnswer(("b", 2)), + // batch 1 + // WM: late record = 0, eviction = 2. Right ("d", 1) is not late, but it is + // evicting in this batch. It still has to be visible to left ("d", 1) from the same + // batch before watermark cleanup removes it. + MultiAddData( + (rightInput, Seq(("d", 1))), + (leftInput, Seq(("d", 1))) + ), + CheckNewAnswer(("d", 1)) + ) + } + } + } + testWithAppendAndUpdate("self left semi join") { outputMode => val (inputStream, query) = setupSelfJoin("left_semi") @@ -2602,6 +2699,36 @@ abstract class StreamingLeftSemiJoinSuite extends StreamingLeftSemiJoinBase { */ } } + + test("left semi join with a range condition requires watermarked bounds on both sides") { + // With only the right side watermarked and a range condition (no watermark in the join keys), + // the left state is never evicted, so never-matched left rows accumulate without bound. This is + // rejected at analysis time. + val (_, _, joined) = setupJoinWithRangeCondition("left_semi", leftWatermark = false) + + val e = intercept[AnalysisException] { + joined.writeStream.format("memory").queryName("leftSemiRightWatermarkOnly") + .outputMode(OutputMode.Append()).start() + } + assert(e.getMessage.contains( + "requires a range bound between watermarked attributes on both sides")) + } + + test("stricter watermark requirements can be disabled for left semi join") { + // With the kill switch off, the right-only range config falls back to the previous behavior and + // is accepted again. + withSQLConf( + SQLConf.STREAMING_JOIN_STRICTER_WATERMARK_REQUIREMENTS_ENABLED.key -> "false") { + val (_, _, joined) = setupJoinWithRangeCondition("left_semi", leftWatermark = false) + val query = joined.writeStream.format("memory") + .queryName("leftSemiFallback").outputMode(OutputMode.Append()).start() + try { + query.processAllAvailable() + } finally { + query.stop() + } + } + } } // Concrete single-mode suites for parallel CI execution and failure isolation. @@ -2638,6 +2765,8 @@ class StreamingFullOuterJoinWithoutVCFSuite extends StreamingFullOuterJoinSuite abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { + import testImplicits._ + test("windowed left anti join") { withTempDir { checkpointDir => val (leftInput, rightInput, joined) = setupWindowedJoin("left_anti") @@ -2828,6 +2957,33 @@ abstract class StreamingLeftAntiJoinBase extends StreamingJoinSuite { CheckNewAnswer(Row(1, 30)) ) } + + test("left anti join suppresses same-batch right row between late and eviction watermarks") { + withTempDir { checkpoint => + withSQLConf(SQLConf.STREAMING_NO_DATA_MICRO_BATCHES_ENABLED.key -> "false") { + val (leftInput, rightInput, joined) = setupEqualityJoinWithZeroWatermark("left_anti") + + testStream(joined, OutputMode.Append())( + StartStream(checkpointLocation = checkpoint.getCanonicalPath), + // batch 0 + // WM: late record = 0, eviction = 0 + MultiAddData( + (leftInput, Seq(("a", 1), ("b", 2))), + (rightInput, Seq(("b", 2), ("c", 1))) + ), + CheckNewAnswer(), + // batch 1 + // WM: late record = 0, eviction = 2. Right ("d", 1) is not late, but it is evicting + // in this batch. It must suppress left ("d", 1) from the same batch before cleanup. + MultiAddData( + (rightInput, Seq(("d", 1))), + (leftInput, Seq(("d", 1))) + ), + CheckNewAnswer(("a", 1)) + ) + } + } + } } abstract class StreamingLeftAntiJoinSuite extends StreamingLeftAntiJoinBase { @@ -2843,7 +2999,7 @@ abstract class StreamingLeftAntiJoinSuite extends StreamingLeftAntiJoinBase { "is not supported in Update output mode, only in Append output mode")) } - test("left anti join with a range condition requires a watermark on the left side") { + test("left anti join with a range condition requires watermarked bounds on both sides") { // Only the right side is watermarked. With a range condition (no watermark in the join keys), // the operator would never build a left-side eviction predicate, so no anti row would ever be // emitted. This must be rejected at analysis rather than silently producing no output. @@ -2854,7 +3010,7 @@ abstract class StreamingLeftAntiJoinSuite extends StreamingLeftAntiJoinBase { .outputMode(OutputMode.Append()).start() } assert(e.getMessage.contains( - "requires a watermark on the left side")) + "requires a range bound between watermarked attributes on both sides")) } } From d22e5a7c83115204046ccf742da59e1aa6e23206 Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Wed, 19 Aug 2026 06:08:41 +0000 Subject: [PATCH 8/9] [SPARK-58611][SS][FOLLOWUP] Add NOT_APPLICABLE binding policy to the new streaming join config spark.sql.streaming.join.stricterWatermarkRequirements.enabled only gates a streaming analyzer validation and never changes a resolved view/UDF/procedure plan, so it uses NOT_APPLICABLE. Required for SparkConfigBindingPolicySuite. --- .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 1 + 1 file changed, 1 insertion(+) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 52576a0d62da2..7fb3485c6df84 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -3796,6 +3796,7 @@ object SQLConf { "sides) are rejected. Set this to false to restore the previous, looser behavior for " + "left semi and left outer joins. Left anti joins always use the stricter requirements.") .version("4.4.0") + .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) .booleanConf .createWithDefault(true) From 5e41c2964fb50f095c61953c3183f8837db58b3c Mon Sep 17 00:00:00 2001 From: Ganesha S Date: Wed, 19 Aug 2026 07:33:33 +0000 Subject: [PATCH 9/9] [SPARK-58611][SS][FOLLOWUP] Use existsJoinedRow for the left anti left-side probe The left-side probe of a left anti join only needs to know whether the right state holds any matching row. Add SymmetricHashJoinStateManager.existsJoinedRow, which short-circuits on the first match and does not update right-side matched flags that left anti never consults, and use it instead of draining getJoinedRows. Also clarify in the docs and analyzer tests that stream-static joins are not stateful and follow the general output-mode rules, so they are allowed in Update mode even when the corresponding stream-stream join is Append-only. --- .../apis-on-dataframes-and-datasets.md | 10 +- .../analysis/UnsupportedOperationsSuite.scala | 14 ++ .../join/StreamingSymmetricHashJoinExec.scala | 62 +++++--- .../join/SymmetricHashJoinStateManager.scala | 59 +++++++ .../SymmetricHashJoinStateManagerSuite.scala | 150 ++++++++++++++++++ 5 files changed, 267 insertions(+), 28 deletions(-) diff --git a/docs/streaming/apis-on-dataframes-and-datasets.md b/docs/streaming/apis-on-dataframes-and-datasets.md index 79cda2115a43c..71ec5b0df1fcd 100644 --- a/docs/streaming/apis-on-dataframes-and-datasets.md +++ b/docs/streaming/apis-on-dataframes-and-datasets.md @@ -1507,8 +1507,10 @@ Additional details on supported joins: - Joins can be cascaded, that is, you can do `df1.join(df2, ...).join(df3, ...).join(df4, ....)`. -- Inner and left semi joins support Append and Update output modes. Left outer, right outer, full - outer, and left anti joins support Append output mode only. Complete output mode is not supported. +- For stream-stream joins, inner and left semi joins support Append and Update output modes. + Stream-stream left outer, right outer, full outer, and left anti joins support Append output mode + only. Complete output mode is not supported for stream-stream joins. Supported stream-static joins + are not stateful and follow the general output-mode rules for non-aggregation queries. - You cannot use mapGroupsWithState and flatMapGroupsWithState before and after joins. @@ -2146,7 +2148,9 @@ Here is the compatibility matrix.
StreamStaticStreamStatic Inner Supported, not stateful
Supported, not stateful
StaticStreamLeft AntiSupported, not stateful
StaticStream Inner Supported, not stateful
Not supported
StreamStreamLeft AntiNot supported
StreamStream Inner Supported, optionally specify watermark on both sides + @@ -1423,6 +1447,13 @@ regarding watermark delays and whether data will be dropped or not. results, optionally specify watermark on left for all state cleanup
Left Anti + Conditionally supported, must specify watermark on right + time constraints for correct + results, optionally specify watermark on left for all state cleanup. Append output mode only +
Left Anti - Conditionally supported, must specify watermark on right + time constraints for correct - results, optionally specify watermark on left for all state cleanup. Append output mode only + Conditionally supported, must watermark the right join key (equality-key joins) or both sides + (range-condition joins), plus time constraints, for correct results; watermarking both sides + on the join event-time column always works. Append output mode only
Left Outer - Conditionally supported, must specify watermark on right + time constraints for correct - results, optionally specify watermark on left for all state cleanup + Conditionally supported. For equality-key joins, watermark both sides of the join key used for + eviction. For range-condition joins, the range bound must use watermarked columns from both + sides.
Left Semi - Conditionally supported, must specify watermark on right + time constraints for correct - results, optionally specify watermark on left for all state cleanup + Conditionally supported. Equality-key joins require a watermark on a join key for state + cleanup. Range-condition joins require watermarked range-bound columns from both sides.
Left Anti - Conditionally supported, must watermark the right join key (equality-key joins) or both sides - (range-condition joins), plus time constraints, for correct results; watermarking both sides - on the join event-time column always works. Append output mode only + Conditionally supported. For equality-key joins, watermark both sides of the join key used for + eviction. For range-condition joins, the range bound must use watermarked columns from both + sides. Append output mode only.
Queries with joinsAppendAppend, Update - Update and Complete mode not supported yet. See the + Update mode is supported only for inner and left semi joins. Complete mode is not supported. + See the support matrix in the Join Operations section for more details on what types of joins are supported. Queries with joins Append, Update - Update mode is supported only for inner and left semi joins. Complete mode is not supported. + For stream-stream joins, Update mode is supported only for inner and left semi joins. + Supported stream-static joins are not stateful and follow the general output-mode rules for + non-aggregation queries. Complete mode is not supported. See the support matrix in the Join Operations section for more details on what types of joins are supported. diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala index 39b0537f1de87..fcd267b06c688 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala @@ -456,6 +456,20 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { Seq("is not supported in Update output mode")) } + // Supported stream-static joins are not stateful and follow the general output-mode rules for + // non-aggregation queries, even when the corresponding stream-stream join is Append-only. + Seq((LeftOuter, "LeftOuter join"), (LeftAnti, "LeftAnti join")).foreach { + case (joinType, name) => + assertSupportedInStreamingPlan( + s"$name with stream-batch relations and update mode", + streamRelation.join(batchRelation, joinType = joinType), + OutputMode.Update()) + } + assertSupportedInStreamingPlan( + "RightOuter join with batch-stream relations and update mode", + batchRelation.join(streamRelation, joinType = RightOuter), + OutputMode.Update()) + // LeftSemi join: Update mode allowed (equivalent to Append mode for non-outer joins) assertSupportedInStreamingPlan( s"LeftSemi join with stream-stream relations and update mode", diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala index 62af3833fc45f..9785c65897cbe 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala @@ -881,23 +881,26 @@ case class StreamingSymmetricHashJoinExec( // If the join type is Left Semi or Left Anti and this is the right side, we can remove // the matched row from the other (left) side's state, since the row won't be produced // anymore for the following input rows. - val joinedRowIter: Iterator[JoinedRow] = if (removeMatchedFromOtherSideState) { - otherSideJoiner.joinStateManager.getJoinedRowsAndRemoveMatched( - key, - thatRow => generateJoinedRow(thisRow, thatRow), - postJoinFilter, - timestampRange = computeTimestampRange(thisRow)).map { row => - numRemovedFromOtherSideDuringJoinCount += 1 - row + val joinedRowIter: Iterator[JoinedRow] = + if (joinType == LeftAnti && joinSide == LeftSide) { + Iterator.empty + } else if (removeMatchedFromOtherSideState) { + otherSideJoiner.joinStateManager.getJoinedRowsAndRemoveMatched( + key, + thatRow => generateJoinedRow(thisRow, thatRow), + postJoinFilter, + timestampRange = computeTimestampRange(thisRow)).map { row => + numRemovedFromOtherSideDuringJoinCount += 1 + row + } + } else { + otherSideJoiner.joinStateManager.getJoinedRows( + key, + thatRow => generateJoinedRow(thisRow, thatRow), + postJoinFilter, + timestampRange = computeTimestampRange(thisRow), + skipUpdatingMatchedFlag) } - } else { - otherSideJoiner.joinStateManager.getJoinedRows( - key, - thatRow => generateJoinedRow(thisRow, thatRow), - postJoinFilter, - timestampRange = computeTimestampRange(thisRow), - skipUpdatingMatchedFlag) - } if (joinType == LeftAnti) { // Left anti join emits nothing while joining, on either side; unmatched left rows are // emitted later during watermark-based eviction of the left side state. The match @@ -905,15 +908,24 @@ case class StreamingSymmetricHashJoinExec( // output iterator: on the left side it drives skip-on-match (a matched left row is not // stored), mirroring left semi. // - // The iterator must be drained fully rather than short-circuited on the first match. - // On the right side it is the `getJoinedRowsAndRemoveMatched` iterator, so draining is - // what removes the matched left rows from state; on the left side draining is required - // by the state manager API. Either way, draining is also needed to detect a match at - // all, since the anti output produces no rows to observe. - var matched = false - while (joinedRowIter.hasNext) { - joinedRowIter.next() - matched = true + val matched = if (joinSide == LeftSide) { + // On the left-side probe path, anti semantics only need to know whether any right + // state row matches. Do not use getJoinedRows here: it may scan all rows for the key + // and update right-side matched flags that left anti never consults. + otherSideJoiner.joinStateManager.existsJoinedRow( + key, + thatRow => generateJoinedRow(thisRow, thatRow), + postJoinFilter, + timestampRange = computeTimestampRange(thisRow)) + } else { + // On the right-side input path, draining is required because the iterator removes all + // matched left rows from state; those rows can no longer become anti output. + var foundMatch = false + while (joinedRowIter.hasNext) { + joinedRowIter.next() + foundMatch = true + } + foundMatch } new AddingProcessedRowToStateCompletionIterator( key, thisRow, Iterator.empty, Some(matched)) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala index 46ef6c3835bab..46a8df46eeb21 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala @@ -83,6 +83,23 @@ trait SymmetricHashJoinStateManager { timestampRange: Option[(Long, Long)] = None, skipUpdatingMatchedFlag: Boolean = false): Iterator[JoinedRow] + /** + * Return whether any joined row for the given key satisfies the provided predicate. + * + * This method does not update matched flags and may stop at the first matching row. It is + * suitable for join paths that only need match existence, such as probing the right side for a + * left anti join input row. + * + * @param timestampRange Optional optimization hint as (minTimestamp, maxTimestamp), both + * inclusive. Derived classes may use it to reduce scan scope but are free to ignore it. + * The predicate must produce correct output regardless of whether this hint is leveraged. + */ + def existsJoinedRow( + key: UnsafeRow, + generateJoinedRow: InternalRow => JoinedRow, + predicate: JoinedRow => Boolean, + timestampRange: Option[(Long, Long)] = None): Boolean + /** * Retrieve all joined rows for the given key and remove the matched rows from state. The joined * rows are generated with the provided generateJoinedRow function and filtered with the provided @@ -519,6 +536,37 @@ class SymmetricHashJoinStateManagerV4( ret.filter(_ != null) } + override def existsJoinedRow( + key: UnsafeRow, + generateJoinedRow: InternalRow => JoinedRow, + predicate: JoinedRow => Boolean, + timestampRange: Option[(Long, Long)]): Boolean = { + def existsInValues(valuesAndMatched: Iterator[ValueAndMatchPair]): Boolean = { + valuesAndMatched.exists { vmp => + predicate(generateJoinedRow(vmp.value)) + } + } + + extractEventTimeFnFromKey(key) match { + case Some(ts) => + existsInValues(keyWithTsToValues.get(key, ts)) + + case _ => + val (minTs, maxTs) = timestampRange.getOrElse((Long.MinValue, Long.MaxValue)) + val valuesByTimestamp = keyWithTsToValues.getValuesInRange(key, minTs, maxTs) + try { + valuesByTimestamp.exists { result => + existsInValues(result.values.iterator) + } + } finally { + valuesByTimestamp match { + case nextIterator: NextIterator[_] => nextIterator.closeIfNeeded() + case _ => + } + } + } + } + override def getJoinedRowsAndRemoveMatched( key: UnsafeRow, generateJoinedRow: InternalRow => JoinedRow, @@ -1270,6 +1318,17 @@ abstract class SymmetricHashJoinStateManagerBase( }.filter(_ != null) } + override def existsJoinedRow( + key: UnsafeRow, + generateJoinedRow: InternalRow => JoinedRow, + predicate: JoinedRow => Boolean, + timestampRange: Option[(Long, Long)]): Boolean = { + val numValues = keyToNumValues.get(key) + keyWithIndexToValue.getAll(key, numValues).exists { keyIdxToValue => + predicate(generateJoinedRow(keyIdxToValue.value)) + } + } + /** Remove using a predicate on keys. */ override def evictByKeyCondition(removalCondition: UnsafeRow => Boolean): Long = { var numRemoved = 0L diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala index acc32c89908ff..036f93201093f 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala @@ -544,6 +544,28 @@ class SymmetricHashJoinStateManagerEventTimeInKeySuite } } + versionsInTest.foreach { ver => + test(s"StreamingJoinStateManager V$ver - existsJoinedRow returns match existence") { + withJoinStateManager( + inputValueAttributes, joinKeyExpressions, stateFormatVersion = ver) { manager => + implicit val mgr = manager + + append(20, 2) + append(20, 3) + + val dummyRow = new GenericInternalRow(0) + assert(manager.existsJoinedRow( + toJoinKeyRow(20), + row => new JoinedRow(row, dummyRow), + jr => jr.getInt(1) == 2)) + assert(!manager.existsJoinedRow( + toJoinKeyRow(20), + row => new JoinedRow(row, dummyRow), + jr => jr.getInt(1) == 99)) + } + } + } + // V1 excluded: V1 converter does not persist matched flags (SPARK-26154) versionsInTest.filter(_ >= 2).foreach { ver => test(s"StreamingJoinStateManager V$ver - matched flag update + eviction roundtrip") { @@ -586,6 +608,47 @@ class SymmetricHashJoinStateManagerEventTimeInKeySuite } } + // V1 excluded: V1 converter does not persist matched flags (SPARK-26154) + versionsInTest.filter(_ >= 2).foreach { ver => + test(s"StreamingJoinStateManager V$ver - existsJoinedRow does not update matched flag") { + withTempDir { checkpointDir => + withJoinStateManagerWithCheckpointDir( + inputValueAttributes, joinKeyExpressions, ver, + checkpointDir, storeVersion = 0, changelogCheckpoint = false) { manager => + implicit val mgr = manager + + append(20, 2) + append(20, 3) + + val dummyRow = new GenericInternalRow(0) + assert(manager.existsJoinedRow( + toJoinKeyRow(20), + row => new JoinedRow(row, dummyRow), + jr => jr.getInt(1) == 2)) + assert(!manager.existsJoinedRow( + toJoinKeyRow(20), + row => new JoinedRow(row, dummyRow), + jr => jr.getInt(1) == 99)) + + mgr.commit() + } + + withJoinStateManagerWithCheckpointDir( + inputValueAttributes, joinKeyExpressions, ver, + checkpointDir, storeVersion = 1, changelogCheckpoint = false) { manager => + implicit val mgr = manager + + val evicted = removeAndReturnByKey(25) + val matchedByValue = evicted.map(p => (toValueInt(p.value), p.matched)).toMap + assert(matchedByValue(2) === false) + assert(matchedByValue(3) === false) + + mgr.commit() + } + } + } + } + // V1 excluded: V1 converter does not persist matched flags (SPARK-26154) versionsInTest.filter(_ >= 2).foreach { ver => test(s"StreamingJoinStateManager V$ver - getJoinedRowsAndRemoveMatched partial") { @@ -914,6 +977,27 @@ class SymmetricHashJoinStateManagerEventTimeInValueSuite } } + versionsInTest.foreach { ver => + test(s"StreamingJoinStateManager V$ver - existsJoinedRow returns match existence") { + withJoinStateManager( + inputValueAttributes, joinKeyExpressions, stateFormatVersion = ver) { manager => + implicit val mgr = manager + + appendAndTest(40, 100, 200, 300) + + val dummyRow = new GenericInternalRow(0) + assert(manager.existsJoinedRow( + toJoinKeyRow(40), + row => new JoinedRow(row, dummyRow), + jr => jr.getInt(1) == 100)) + assert(!manager.existsJoinedRow( + toJoinKeyRow(40), + row => new JoinedRow(row, dummyRow), + jr => jr.getInt(1) == 999)) + } + } + } + // V1 excluded: V1 converter does not persist matched flags (SPARK-26154) versionsInTest.filter(_ >= 2).foreach { ver => test(s"StreamingJoinStateManager V$ver - matched flag update + eviction roundtrip") { @@ -952,6 +1036,47 @@ class SymmetricHashJoinStateManagerEventTimeInValueSuite } } + // V1 excluded: V1 converter does not persist matched flags (SPARK-26154) + versionsInTest.filter(_ >= 2).foreach { ver => + test(s"StreamingJoinStateManager V$ver - existsJoinedRow does not update matched flag") { + withTempDir { checkpointDir => + withJoinStateManagerWithCheckpointDir( + inputValueAttributes, joinKeyExpressions, ver, + checkpointDir, storeVersion = 0, changelogCheckpoint = false) { manager => + implicit val mgr = manager + + appendAndTest(40, 100, 200, 300) + + val dummyRow = new GenericInternalRow(0) + assert(manager.existsJoinedRow( + toJoinKeyRow(40), + row => new JoinedRow(row, dummyRow), + jr => jr.getInt(1) == 100, + timestampRange = Some((100L, 100L)))) + assert(!manager.existsJoinedRow( + toJoinKeyRow(40), + row => new JoinedRow(row, dummyRow), + jr => jr.getInt(1) == 999, + timestampRange = Some((100L, 100L)))) + + mgr.commit() + } + + withJoinStateManagerWithCheckpointDir( + inputValueAttributes, joinKeyExpressions, ver, + checkpointDir, storeVersion = 1, changelogCheckpoint = false) { manager => + implicit val mgr = manager + + val evicted = removeAndReturnByValue(125) + val matchedByValue = evicted.map(p => (toValueInt(p.value), p.matched)).toMap + assert(matchedByValue(100) === false) + + mgr.commit() + } + } + } + } + // V1 excluded: V1 converter does not persist matched flags (SPARK-26154) versionsInTest.filter(_ >= 2).foreach { ver => test(s"StreamingJoinStateManager V$ver - getJoinedRowsAndRemoveMatched partial") { @@ -1096,6 +1221,31 @@ class SymmetricHashJoinStateManagerEventTimeInValueSuite } } + test("StreamingJoinStateManager V4 - existsJoinedRow with timestampRange") { + withJoinStateManager( + inputValueAttributes, joinKeyExpressions, stateFormatVersion = 4) { manager => + implicit val mgr = manager + val dummyRow = new GenericInternalRow(0) + + Seq(10, 20, 30, 40, 50).foreach(append(40, _)) + + def existsInRange(range: Option[(Long, Long)]): Boolean = { + manager.existsJoinedRow( + toJoinKeyRow(40), + row => new JoinedRow(row, dummyRow), + _ => true, + timestampRange = range) + } + + assert(existsInRange(Some((20L, 40L)))) + assert(existsInRange(Some((20L, 20L)))) + assert(existsInRange(Some((25L, 35L)))) + assert(!existsInRange(Some((60L, 100L)))) + assert(!existsInRange(Some((0L, 5L)))) + assert(existsInRange(None)) + } + } + test("StreamingJoinStateManager V4 - getJoinedRowsAndRemoveMatched with timestampRange") { withJoinStateManager( inputValueAttributes, joinKeyExpressions, stateFormatVersion = 4) { manager =>