[slachiewicz]

Closes #4267

Rationale for this change

The spec's column projection rules fill a field that is missing from a data file from the identity partition value in the manifest. ArrowProjectionVisitor.struct applied that rule only inside the branch for optional fields and fields with an initial-default, so a required identity partition column raised ResolveError instead. Tables whose writers keep partition columns out of the data files, such as Hive-style layouts and tables converted from Delta, hit this on every read.

A required field now takes the projected value. A null partition value raises ResolveError for a required field, even when the field has an initial-default, unless the field's parent struct is null in every row.

Are these changes tested?

Yes. test_identity_transform_column_projection_required_field covers the projected value, the column's non-nullable type and row filters on it, with and without an initial-default, plus the null partition value. test_identity_transform_column_projection_required_field_under_null_parent covers a required child of an optional struct. Both fail without the change.

Are there any user-facing changes?

Scans of such tables return the partition value instead of raising ResolveError.

[Copilot]

🟡 Changes recommended

Null partition handling remains incorrect for required fields with defaults and for required children masked by null optional parents.

2 open issues

• Allow omitted required children beneath null optional parents · New
• Reject null partition values despite a non-null initial default · New

What changed in this PR

This PR addresses #4267 by allowing PyIceberg scans to populate required columns omitted from data files using non-null identity partition values.

Changes:

• Extends Arrow projection eligibility to required fields with non-null partition values.
• Adds regression coverage for projected values, non-nullable output, row filtering, and null rejection without a default.

File	Description
tests/​io/​test_pyarrow.py	Adds required identity partition column projection tests.
pyiceberg/​io/​pyarrow.py	Allows non-null partition values to populate missing required fields.

_{Review effort: Balanced}