[yunchipang]

Which issue does this PR close?

Closes #7033.

Rationale for this change

The native Iceberg scan failed with Incorrect number of arrays for StructArray fields, expected 2 got 1 when a data file omitted an optional nested field without a schema change. The iceberg-rust bump in #7028 fixed this, since iceberg-rust now matches nested fields by field id and fills in missing ones with NULLs. Nothing tested this case.

What changes are included in this PR?

A test in CometIcebergNativeSuite that writes a data file without s.b and l.element.y, using write.spark.accept-any-schema. It then checks that the native scan matches Spark, with nested schema pruning on and off.

How are these changes tested?

The new test passes on main. On the parent of #7028, it fails with the error from the issue.

[sunchao]

The new regression passed on Spark 4.1.3 with Iceberg 1.11. I inspected the Parquet footers produced by the same fixture and confirmed that the narrow files omit both nested leaves. One assertion would preserve that coverage as the writer changes.

[sunchao]

Could the test check that at least one appended Parquet file lacks both s.b and the list element’s y leaf? If a writer upgrade materializes them as all-null columns, both Spark comparisons and the native-plan assertion will still pass, and the test will stop exercising #7033. This suite already reads file_path from the Iceberg files table and opens its footer with ParquetFileReader.

[yunchipang]

thanks! added a footer check.

[sunchao]

Reviewed 801755b with five independent scopes. The new assertions first verify the nested leaf paths against a full-schema file and then require a physical file omitting both leaves, so they preserve the intended coverage as the writer changes. I compiled the exact updated suite and ran this regression on Spark 4.1.3 / Iceberg 1.11; it passed with the native scan assertions. The production sources are unchanged from the previously tested revision. No remaining actionable issue found.

[manuzhang]

@yunchipang Please remove these lines from the doc.

datafusion-comet/docs/source/user-guide/latest/iceberg.md

Lines 832 to 835 in 0f7f8e4

	- **A data file that omits an optional nested field.** The native scan fails on a data file that
	lacks an optional struct field, which a write with `write.spark.accept-any-schema=true` can
	produce without any schema change
	([#7033](https://github.com/apache/datafusion-comet/issues/7033)).

[manuzhang]

@yunchipang Meanwhile, could you please enable Github Actions on your fork repository such that CI can run there?

[comphead]

Thanks @yunchipang, the footer check makes this a solid regression test for #7033.

[comphead]

This is the third copy in this suite of opening a footer with ParquetFileReader.open(HadoopInputFile.fromPath(new Path(...), spark.sessionState.newHadoopConf())) and closing it in a finally, after the page-index test around line 1024 and the INT96 test around line 4653. Each uses the reader differently, so a small private withParquetReader[T](path: String)(f: ParquetFileReader => T): T would cover all three and keep the fully qualified names in one place. Would you add it here?

[yunchipang]

thanks for the review! added the share reader at line 169

[andygrove]

Thanks @yunchipang. The footer assertions are what make this a useful regression test for #7033: they check the leaf paths against the full-schema file first, then require a file that has neither leaf, so a writer change that fills in the missing fields would fail the test instead of quietly passing. The withParquetReader helper also cleans up the other two footer reads nicely.