Skip to content
Merged
4 changes: 4 additions & 0 deletions docs/source/user-guide/latest/compatibility/scans.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,10 @@ and `INT32 → DOUBLE` widening that Spark 4.0+ accepts unconditionally; `Timest
is rejected by Spark 3.x but accepted by Spark 4.0+). Comet aims to follow the per-version Spark
behavior.

- **List conversion error paths assume Spark's standard encoding**. Comet inserts `list`
before the element name when reporting a rejected array element conversion. Arrow's schema
omits the repeated group name, so paths for legacy LIST encodings or custom group names may
differ from Spark's Parquet column path.
- **`ParquetSchemaConvert` errors do not include the file path**. The mismatch itself is detected and
rejected correctly, but the resulting Spark error message reads
`Encountered error while reading file . Data type mismatches…` (note the empty path). Behavior is
Expand Down
3 changes: 2 additions & 1 deletion native/core/src/execution/planner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6802,7 +6802,8 @@ mod tests {
*/
#[tokio::test]
async fn test_nested_types_list_of_struct_by_index() -> Result<(), DataFusionError> {
let test_data = "select make_array(named_struct('a', 1, 'b', 'n', 'c', 'x')) c0";
let test_data =
"select make_array(named_struct('a', cast(1 as int), 'b', 'n', 'c', 'x')) c0";

// Define schema Comet reads with
let required_schema = Schema::new(Fields::from(vec![Field::new(
Expand Down
Loading
Loading