str_to_map does not honour mapKeyDedupPolicy=LAST_WIN

[peterxcli]

Describe the bug

With spark.sql.mapKeyDedupPolicy=LAST_WIN, str_to_map raises [DUPLICATED_MAP_KEY] on an input string that repeats a key, while Spark returns a map holding the last value for that key. Comet fails the query where Spark succeeds.

The error also surfaces as a raw native failure rather than Spark's SparkRuntimeException, because nothing translates it on the JVM side.

Steps to reproduce

SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
SELECT str_to_map('a:1,b:2,a:3');

Spark returns {a -> 3, b -> 2}. Comet raises:

[DUPLICATED_MAP_KEY] Duplicate map key 'a' was found, please check the input data.
To allow duplicate keys with last-value-wins semantics, set
`datafusion.spark.map_key_dedup_policy` to `LAST_WIN`.

The default EXCEPTION policy is unaffected: both engines raise on a duplicate key there, so this only shows up once the policy is changed.

Expected behavior

str_to_map follows spark.sql.mapKeyDedupPolicy the way Spark's ArrayBasedMapBuilder does, keeping the last value for each duplicate key under LAST_WIN.

Additional context

The native kernel already implements this. datafusion-spark's SparkStrToMap reads datafusion.spark.map_key_dedup_policy (see datafusion_common::config::SparkOptions), which takes the same EXCEPTION and LAST_WIN values as the Spark config. Two things stop the Spark setting from reaching it:

1. spark.sql.mapKeyDedupPolicy never crosses JNI. CometExecIterator.serializeCometSQLConfs sends only keys under spark.comet., plus a short explicit list.
2. create_scalar_function_expr in native/core/src/execution/planner.rs hands every ScalarFunctionExpr a fresh ConfigOptions::default(), so a kernel reading a session option sees DataFusion's defaults rather than the session's values.

CometStrToMap.getSupportLevel does not inspect the policy either, so the expression stays Compatible and takes the native path under LAST_WIN instead of routing through the JVM codegen dispatcher.

spark/src/test/resources/sql-tests/expressions/map/str_to_map.sql carries a matching note:

-- TODO: Add LAST_WIN policy tests when spark.sql.mapKeyDedupPolicy config is supported

#5854 fixes both points above while closing #4680, and adds str_to_map_dedup_policy.sql covering LAST_WIN. This issue records the divergence on its own, since it reaches a different expression from the one #4680 and #5589 describe and would otherwise go untracked if that PR is split up.

Related: #4680 (null keys in map_from_arrays / map_from_entries), #5589 (map_from_arrays fallback under LAST_WIN).

[peterxcli]

take