Repository navigation
pyarrow.json.read_json ignores nullable=True on fields with non-nullable subfields in explicit_schema parse_options #33055
Description
Summary:
The parser seems to be ignoring the "nullable" flag on the parent field.
This behavior may be related to ARROW-16603, but that issue covers the opposite case - failing to include the "not null" constraints in the schema of the output table.
Reproducible example:
import pyarrow as pa
interior_struct = pa.struct([
pa.field(name='some_information', type=pa.int64(), nullable=False)
])
sample_schema = pa.schema([
pa.field(name='my_struct', type=interior_struct, nullable=True)
])
# The issue persists regardless of the input JSON,
# so long as there is no field named "my_struct" in the input.
# Using an empty JSON object for example
sample_json_bytes = io.BytesIO('{}'.encode())
table = pa.json.read_json(
input_file=sample_json_bytes,
parse_options=pa.json.ParseOptions(explicit_schema=sample_schema)
)
print(table)Expected output:
Table containing one column:
- Name is "my_struct"
- Type is "interior_struct"
- Contains a single value, which is null
Actual output:
ArrowInvalid: JSON parse error: a required field was nullReporter: Ryan Weisman
Note: This issue was originally created as ARROW-17835. Please see the migration documentation for further details.
Activity
thisisnic commentedon Oct 26, 2026
Related to #40681 and #31957.