GH-47417: [C++] Fix JSON parser losing nulls in a null-typed list - #51108
GH-47417: [C++] Fix JSON parser losing nulls in a null-typed list#51108advitrocks9 wants to merge 2 commits into
Conversation
|
|
There was a problem hiding this comment.
Pull request overview
Fixes a long-standing JSON parsing bug in the C++ BlockParser where null elements could be silently dropped (or trigger DCHECKs in debug) when appending nulls to a null-typed list, by ensuring the inline null-count state in BuilderPtr is updated in-place.
Changes:
- Change
RawBuilderSet::AppendNullto take aBuilderPtr*so increments to the inline null-count forKind::kNullpersist to the caller. - Update all
AppendNullcall sites to pass builder pointers accordingly (including nested object-field null propagation). - Add a regression test covering nulls inside lists (null-only and promoted-from-null cases).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| cpp/src/arrow/json/parser.cc | Fixes null appends by mutating the BuilderPtr in-place, preventing lost null counts in null-typed list builders. |
| cpp/src/arrow/json/parser_test.cc | Adds a regression test to ensure nulls inside lists are preserved and don’t break list offsets. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| TEST(BlockParser, NullsInList) { | ||
| auto options = ParseOptions::Defaults(); | ||
| options.unexpected_field_behavior = UnexpectedFieldBehavior::InferType; | ||
| AssertParseColumns(options, R"({"a": [null, null], "b": [null, "hi", null]})", | ||
| {field("a", list(null())), field("b", list(utf8()))}, | ||
| {"[[null, null]]", R"([[null, "hi", null]])"}); | ||
| } |
There was a problem hiding this comment.
Added a test for the explicit schema under Error and Ignore.
|
|
There was a problem hiding this comment.
🟢 Approval recommended
The fix is narrowly scoped, updates all relevant call sites, and adds targeted regression tests exercising both the all-null and promotion scenarios described.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
Rationale for this change
{"a": [null, 1]}parses to[[1]]. Nothing raises, andvalidate(full=True)is the only thingthat notices:
Debug builds abort on a DCHECK instead, at
parser.cc:540if the second element is null andparser.cc:1151if it is typed.What changes are included in this PR?
RawBuilderSet::AppendNulltakes the child builder by value. Builders ofKind::kNullhave noarena, their null count is packed into
BuilderPtr::index, so the increment updates the parentbut is lost to the caller. The next null then re-increments the same stale value, which leaves an
all-null list as
NullArray(1)and a promoted list with its leading nulls missing.It now takes a
BuilderPtr*, matchingMakeBuilderin the same class. The regression came inwith
b7054c21aaband has shipped since 0.14.0.Are these changes tested?
BlockParser.NullsInListcovers both arms, andBlockParserWithSchema.NullsInListruns anexplicit
list(null())schema underErrorandIgnore.arrow-json-testpasses 262/262 inDebug and Release. Reverting
parser.ccwith the tests in place aborts both in Debug and failsboth on null count in Release.
Are there any user-facing changes?
Those inputs parse correctly now. This is not inference-only: an explicit
list(null())schemareaches the same arm under both
ErrorandIgnore.This PR contains a "Critical Fix". Well-formed JSON silently loses list elements in release
builds and aborts on a DCHECK in debug builds.