Skip to content

fix: get_json_object returns first value for duplicate keys to match Spark#4971

Open
u70b3 wants to merge 2 commits into
apache:mainfrom
u70b3:fix/json-dup-key-first-wins
Open

fix: get_json_object returns first value for duplicate keys to match Spark#4971
u70b3 wants to merge 2 commits into
apache:mainfrom
u70b3:fix/json-dup-key-first-wins

Conversation

@u70b3

@u70b3 u70b3 commented Jul 18, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #4947.

Rationale for this change

For a JSON object with duplicate keys, Spark's GetJsonObjectEvaluator stops at the first matching field, but Comet's native implementation resolved the key to its last occurrence (matching serde_json's overwrite semantics rather than Spark's):

SELECT get_json_object('{"a":1,"a":2}', '$.a');
-- Spark: 1
-- Comet (before this PR): 2

What changes are included in this PR?

SegmentVisitor::visit_map in native/spark-expr/src/string_funcs/get_json_object.rs now locks in the first key match and consumes later occurrences via IgnoredAny instead of re-parsing them with PathSeed:

  • The lock is keyed on the key match, not on a successful subpath resolution: if the first matching value does not contain the rest of the path, the result is null and the second duplicate key is never consulted, matching Spark.
  • All entries are still visited, so the existing malformed/trailing-garbage rejection behavior is unchanged.

How are these changes tested?

  • Flipped the existing test_duplicate_key_last_wins unit test to test_duplicate_key_first_wins.
  • Added test_duplicate_key_first_wins_nested ({"a":{"b":1},"a":{"b":2}} / $.a.b -> 1).
  • Added test_duplicate_key_first_match_locks_value ({"a":{"x":1},"a":{"b":2}} / $.a.b -> null, matching Spark's lock-on-first-match semantics).
  • Full datafusion-comet-spark-expr suite passes (497 tests), plus clippy and fmt checks.
  • Checked spark/src/test/resources/sql-tests/ and CometJsonJvmSuite for duplicate-key cases: the only repeated keys there are across different objects inside arrays, which are unaffected by this change.

@u70b3
u70b3 force-pushed the fix/json-dup-key-first-wins branch from bf2cf17 to 90db735 Compare July 18, 2026 00:46
…h Spark

Spark's GetJsonObjectEvaluator stops at the first matching field, but
Comet's SegmentVisitor.visit_map kept overwriting the result, resolving
a duplicated key to its last occurrence. Lock in the first match (even
when the subpath misses, per Spark semantics) and skip later occurrences
while still consuming all entries to validate the document.

Closes apache#4947
@u70b3
u70b3 force-pushed the fix/json-dup-key-first-wins branch from 90db735 to ec8fafb Compare July 18, 2026 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native implementation of get_json_object returns last value for duplicate keys, Spark returns first

1 participant