[SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity - #58222
Open
szehon-ho wants to merge 1 commit into
Open
[SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity#58222szehon-ho wants to merge 1 commit into
szehon-ho wants to merge 1 commit into
Conversation
… case sensitivity SPARK-58517 made Declarative Pipelines honor spark.sql.caseSensitive when inferring and evolving pipeline table schemas, but described the change as affecting only unreleased branches, so no migration note was added. The previous case-sensitive-only behavior did ship in 4.1.x and 4.2.x, and the change also introduces a new error condition when flows disagree on the effective value.
anew
approved these changes
Aug 22, 2026
anew
left a comment
anew
left a comment
Contributor
There was a problem hiding this comment.
This looks good to me. Thanks @szehon-ho !
uros-b
approved these changes
Aug 22, 2026
Member
|
+1, thank you @szehon-ho and @anew! |
AnishMahto
approved these changes
Aug 22, 2026
| - Since Spark 4.3, the Spark-recognized time-travel options (`versionAsOf` and `timestampAsOf`, or the keys configured by `spark.sql.timeTravelVersionKey` and `spark.sql.timeTravelTimestampKey`) are rejected with `UNSUPPORTED_FEATURE.TIME_TRAVEL` on catalog-backed Data Source V2 writes, including table creation and replacement through `DataFrameWriterV2`, because writes must target the current table state rather than a historical version. Previously, Spark passed them to the connector as ordinary write options. | ||
| - Since Spark 4.3, `HAVING` is evaluated before window functions when the `SELECT` list also contains generator functions such as `explode`. Previously, window functions could include groups removed by `HAVING` and produce incorrect results. | ||
| - Since Spark 4.3, the Spark Connect session errors `INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` carry SQLSTATE `08003` instead of `HY000`; the condition names are unchanged. Code matching these errors on SQLSTATE should match `08003` or class `08`. | ||
| - Since Spark 4.3, [Declarative Pipelines](declarative-pipelines-programming-guide.html) honors `spark.sql.caseSensitive` when inferring and evolving pipeline table schemas. Under case-insensitive resolution (the default), column names that differ only in case now identify the same column: flows writing to one table contribute a single column rather than one per spelling, and a column that differs only in case from one already persisted in the target is written to that column instead of being added alongside it. Previously such names were always treated as distinct, producing a table schema that Spark's own resolver could not disambiguate and that could fail later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`. When two flows' columns fold together but their types are incompatible, the update now fails at validation with `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA`. Where the spellings differ, the surviving one comes from the flow with the lowest identifier, or, when the target table already has the column, its persisted spelling. Set `spark.sql.caseSensitive` to `true` to keep names differing only in case distinct, as before. |
Contributor
There was a problem hiding this comment.
Optional since maybe its unnecessarily specific:
when the target table already has the column
Only applies to incremental target tables (STs) and not MVs, which re-infer schema every update. And if a table defined in the pipeline explicitly defines its schema, the spelling there is always respected over the inferred spelling from flows.
dongjoon-hyun
approved these changes
Aug 23, 2026
HyukjinKwon
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds two entries to the "Upgrading from Spark SQL 4.2 to 4.3" section of
docs/sql-migration-guide.md, documenting the behavior change made bySPARK-58517 (#57722):
spark.sql.caseSensitivewhen inferring andevolving pipeline table schemas. Under case-insensitive resolution (the
default), column names differing only in case identify the same column, so
flows contribute a single column rather than one per spelling, and the note
states which spelling survives.
spark.sql.caseSensitive, or the update fails with the newCONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITYerror condition.Documentation only; no code change.
Why are the changes needed?
SPARK-58517 first ships in 4.3.0 (merged to
master,branch-4.x, andbranch-4.3), and the behavior it changed has been in place since 4.1.0:SchemaMergingUtils.mergeSchemasmerged positionally, leavingStructType.merge'scaseSensitive = truedefault;DataflowGraph.inferredSchemamerged with
reduce(SchemaMergingUtils.mergeSchemas); andSchemaInferenceUtils.diffSchemaskeyed columns on exact field names. Usersupgrading from 4.1.x or 4.2.x will see two observable differences that a
migration note should cover:
later with errors such as
COLUMN_ALREADY_EXISTSorAMBIGUOUS_REFERENCE,now folds into a single column.
spark.sql.caseSensitivenow fails upfront with
CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY.Does this PR introduce any user-facing change?
No. This is a documentation-only change. It documents a user-facing change that
was already made by SPARK-58517.
How was this patch tested?
No tests; documentation only. The error condition names in the notes were
checked against the tree (
CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITYinerror-conditions.json,UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMAinGraphErrors.scala), as was the "lowest flow identifier wins" rule(
SchemaInferenceUtils.inferSchemaFromFlowssorts by_.identifier.unquotedString). The link formdeclarative-pipelines-programming-guide.htmlmatches the existing link indocs/index.md.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5 (Cursor)