Native Parquet scan reads invalid UTF-8 in nested BINARY values as unvalidated strings on DataFusion 56 #7469

Describe the bug

On DataFusion 56, the native Parquet scan decodes a nested Parquet BINARY value that is read as a string straight into an Arrow Utf8 array, without checking that it is valid UTF-8. Nested values here means list elements and struct fields. An invalid value then gives a StringArray that holds invalid UTF-8. In a debug build (and the CI profile, which enables debug assertions) the parquet reader panics. In a release build the array is unsound for every string kernel that reads it.

On current main, which is on DataFusion 55.2, #7106 decodes these values with Spark's lossy decode, so an ill-formed sequence becomes U+FFFD. The DataFusion 56 upgrade (#6410, draft PR #7249) undoes that.

nested_arrow_binary_types_read_as_string_decode_invalid_utf8 in native/core/src/parquet/parquet_exec.rs shows it. A list of [b"one", [0xff]] read as array<string> panics in parquet/src/arrow/buffer/offset_buffer.rs:139:

InvalidArgumentError("Invalid UTF8 sequence at string index 1 (3..4): invalid utf-8 sequence of 1 bytes from index 0")

It panics for plain Binary elements too, not only the Arrow-specific LargeBinary and BinaryView, so files written by Spark are affected whenever a nested BINARY column is read as a string, through a read schema or spark.sql.parquet.binaryAsString.

apache/datafusion#25193 made apply_file_schema_type_coercions recurse into nested types. When the table schema has Utf8 where the file has Binary, LargeBinary or BinaryView, DataFusion now rewrites the nested file field to Utf8 and hands that schema to the parquet reader. DataFusion 55.2 did this only for top-level fields, so nested values reached Comet's parquet_convert_array as binary and #7106's decode.

The parquet reader validates UTF-8 only when the column carries the UTF8 annotation (validate_utf8 = desc.converted_type() == ConvertedType::UTF8 in parquet/src/arrow/array_reader/byte_array.rs), which an unannotated BINARY column does not. DataFusion's own docs on coerce_data_type note that the conversion "hands back an unvalidated string array".

This is the nested form of #7032's top-level gap, which already exists on main.

Steps to reproduce

Run nested_arrow_binary_types_read_as_string_decode_invalid_utf8 on the datafusion-56 branch.

Expected behavior

An ill-formed sequence in a nested BINARY value read as a string decodes to U+FFFD, as it does on main.

Additional context

• Possible fixes: have the parquet reader validate UTF-8 when the target Arrow type is a string, whatever the annotation; let callers opt out of the binary-to-string coercion in DataFusion; or keep nested binary file types out of the coercion in Comet and convert them with Comet's decoder.
• Upstream issue: REGRESSION: Parquet scan returns unvalidated Utf8 for nested BINARY fields read as strings (regression from #25193) datafusion#27065, fix in draft fix: keep nested Parquet binary fields out of the binary-to-string coercion datafusion#27067 (keeps nested binary fields out of the coercion). The full fix, for top-level (Native Parquet scan returns NULL for nested BINARY values that are not valid UTF-8 when they are read as STRING #7032) and nested values alike, is the opt-in InvalidUtf8::Replace reader option proposed in Parquet reader: option to replace invalid UTF-8 instead of failing arrow-rs#12047, provided it checks UTF-8 whenever the target type is a string. With it, Comet's native scan would decode invalid UTF-8 to U+FFFD at every level. On DataFusion 55.2.0 the same read without Comet fails with a UTF-8 error; on branch-56 it panics in debug builds and returns an invalid StringArray in release builds.
• Until a fix lands, the test is ignored on the DataFusion 56 branch.

Willingness to contribute

I can contribute a fix for this bug independently