[SPARK-56955][CONNECT] Replace -Dio.netty.noUnsafe=false with --sun-misc-unsafe-memory-access=allow for a more complete JDK 25+ support by attilapiros · Pull Request #56006 · apache/spark

attilapiros · 2026-05-20T01:53:25Z

What changes were proposed in this pull request?

Avoiding the java.lang.UnsupportedOperationException for spark-shell --remote on JDK25.

This PR is simply replaces -Dio.netty.noUnsafe=false with --sun-misc-unsafe-memory-access=allow.
This new flag does not cause any issue for earlier JDK versions as -XX:+IgnoreUnrecognizedVMOptions is already used.

Why are the changes needed?

On JDK 25 when spark-shell is started with --remote flag we get an exception at the start:

$ ./bin/spark-shell --remote sc://localhost:15002
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
26/05/19 17:38:31 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
26/05/19 17:38:31 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type
26/05/19 17:38:31 INFO CheckAllocator: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class
Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.sparkproject.org.apache.arrow.memory.netty.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at java.base/java.lang.Class.forName(Class.java:458)
	at org.sparkproject.org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:105)
	at org.sparkproject.org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:92)
	at org.sparkproject.org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:826)
	at org.sparkproject.org.apache.arrow.memory.ImmutableConfig.access$001(ImmutableConfig.java:20)
	at org.sparkproject.org.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:80)
	at org.sparkproject.org.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:43)
	at org.sparkproject.org.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:492)
	at org.sparkproject.org.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:72)
	at org.apache.spark.sql.connect.SparkSession.<init>(SparkSession.scala:89)
	at org.apache.spark.sql.connect.SparkSession$Builder.tryCreateSessionFromClient(SparkSession.scala:1059)
	at org.apache.spark.sql.connect.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:1119)
	at org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:949)
	at org.apache.spark.sql.connect.SparkSession$Builder.getOrCreate(SparkSession.scala:1118)
	at org.apache.spark.sql.application.ConnectRepl$.$anonfun$doMain$1(ConnectRepl.scala:91)
	at org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:949)
	at org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:68)
	at org.apache.spark.sql.application.ConnectRepl$.main(ConnectRepl.scala:58)
	at org.apache.spark.sql.application.ConnectRepl.main(ConnectRepl.scala)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1033)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:226)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:95)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1171)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1180)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.UnsupportedOperationException
	at org.sparkproject.io.netty.buffer.EmptyByteBuf.memoryAddress(EmptyByteBuf.java:961)
	at org.sparkproject.io.netty.buffer.DuplicatedByteBuf.memoryAddress(DuplicatedByteBuf.java:115)
	at org.sparkproject.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:45)
	at org.sparkproject.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:47)
	at org.sparkproject.org.apache.arrow.memory.netty.NettyAllocationManager.<clinit>(NettyAllocationManager.java:54)
	... 32 more

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Running the spark-shell --remote locally and AmmoniteReplE2ESuite on JDK 25 as:

./build/mvn test  -pl sql/connect/client/jvm -Dsuites="*.AmmoniteReplE2ESuite"

And also testing all the ArrowConvertersSuite tests:

$ build/sbt "sql/testOnly *ArrowConvertersSuite"
...
[info] Run completed in 4 seconds, 719 milliseconds.
[info] Total number of tests run: 43
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 43, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 127 s (0:02:07.0), completed May 20, 2026, 6:31:55 AM
$ java --version
openjdk 25.0.3 2026-04-21 LTS
OpenJDK Runtime Environment Temurin-25.0.3+9 (build 25.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-25.0.3+9 (build 25.0.3+9-LTS, mixed mode, sharing)

Was this patch authored or co-authored using generative AI tooling?

No.

yadavay-amzn

LGTM. Leveraging the existing IgnoreUnrecognizedVMOptions to avoid version gating is the right call. Verified the same fix works locally on Corretto 25.0.3.

pan3793 · 2026-05-20T08:18:23Z

The vanilla netty lib Java 25 support has been addressed by adding -Dio.netty.noUnsafe=false in SPARK-55670 (#54470), but your stacktrace indicates the error is thrown from a shaded netty lib org.sparkproject.io.netty

according to netty/netty#14942 (comment), we can use --sun-misc-unsafe-memory-access=true and remove -Dio.netty.noUnsafe=false, this works for both vanilla and shaded netty libs. so please update all places as the opts spread in many places.

You can either ignore the warning, since Netty is auto-detecting availability, or you can disable it explicitly with -Dio.netty.noUnsafe=true (note, this property have have a different name in the shaded version).

You can also control this behavior with the --sun-misc-unsafe-memory-access argument:
    --sun-misc-unsafe-memory-access=<value>
                     allow or deny usage of unsupported API sun.misc.Unsafe
                     <value> is one of "allow", "warn", "debug", or "deny".
                     The default value is "warn".

…ess=allow

attilapiros · 2026-05-20T13:37:49Z

Thanks @pan3793! I have done all the replacements.
Tested it with the:

$ build/sbt "sql/testOnly *ArrowConvertersSuite"
...
[info] Run completed in 4 seconds, 719 milliseconds.
[info] Total number of tests run: 43
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 43, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 127 s (0:02:07.0), completed May 20, 2026, 6:31:55 AM
$ java --version
openjdk 25.0.3 2026-04-21 LTS
OpenJDK Runtime Environment Temurin-25.0.3+9 (build 25.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-25.0.3+9 (build 25.0.3+9-LTS, mixed mode, sharing)

attilapiros · 2026-05-20T15:45:09Z

When the tests finished I can merge this and even backport it up to 4.2.0

dongjoon-hyun

+1, LGTM. Thank you, @attilapiros and all.

dongjoon-hyun · 2026-05-20T15:50:18Z

cc @huaxingao

dongjoon-hyun

BTW, @attilapiros , it would be great to revise the PR title about the PR code itself. The PR description looks good. Only the title seems to need revisions.

JIRA title is focusing on the issue reporting.
The PR title (and commit log) is focusing on summarizing what is the code change)

attilapiros · 2026-05-20T16:59:40Z

@dongjoon-hyun What about "Replace -Dio.netty.noUnsafe=false with --sun-misc-unsafe-memory-access=allow for a more complete JDK 25+ support" ?

dongjoon-hyun · 2026-05-20T17:32:46Z

Thank you. Sounds better to me.

…isc-unsafe-memory-access=allow for a more complete JDK 25+ support ### What changes were proposed in this pull request? Avoiding the `java.lang.UnsupportedOperationException` for `spark-shell --remote` on JDK25. This PR is simply replaces `-Dio.netty.noUnsafe=false` with `--sun-misc-unsafe-memory-access=allow`. This new flag does not cause any issue for earlier JDK versions as `-XX:+IgnoreUnrecognizedVMOptions` is already used. ### Why are the changes needed? On JDK 25 when `spark-shell` is started with `--remote` flag we get an exception at the start: ``` $ ./bin/spark-shell --remote sc://localhost:15002 WARNING: Using incubator modules: jdk.incubator.vector WARNING: package sun.security.action not in java.base 26/05/19 17:38:31 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true. 26/05/19 17:38:31 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type 26/05/19 17:38:31 INFO CheckAllocator: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class Exception in thread "main" java.lang.ExceptionInInitializerError at org.sparkproject.org.apache.arrow.memory.netty.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467) at java.base/java.lang.Class.forName(Class.java:458) at org.sparkproject.org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:105) at org.sparkproject.org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:92) at org.sparkproject.org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:826) at org.sparkproject.org.apache.arrow.memory.ImmutableConfig.access$001(ImmutableConfig.java:20) at org.sparkproject.org.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:80) at org.sparkproject.org.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:43) at org.sparkproject.org.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:492) at org.sparkproject.org.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:72) at org.apache.spark.sql.connect.SparkSession.<init>(SparkSession.scala:89) at org.apache.spark.sql.connect.SparkSession$Builder.tryCreateSessionFromClient(SparkSession.scala:1059) at org.apache.spark.sql.connect.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:1119) at org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:949) at org.apache.spark.sql.connect.SparkSession$Builder.getOrCreate(SparkSession.scala:1118) at org.apache.spark.sql.application.ConnectRepl$.$anonfun$doMain$1(ConnectRepl.scala:91) at org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:949) at org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:68) at org.apache.spark.sql.application.ConnectRepl$.main(ConnectRepl.scala:58) at org.apache.spark.sql.application.ConnectRepl.main(ConnectRepl.scala) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) at java.base/java.lang.reflect.Method.invoke(Method.java:565) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1033) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:226) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:95) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1171) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1180) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.UnsupportedOperationException at org.sparkproject.io.netty.buffer.EmptyByteBuf.memoryAddress(EmptyByteBuf.java:961) at org.sparkproject.io.netty.buffer.DuplicatedByteBuf.memoryAddress(DuplicatedByteBuf.java:115) at org.sparkproject.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:45) at org.sparkproject.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:47) at org.sparkproject.org.apache.arrow.memory.netty.NettyAllocationManager.<clinit>(NettyAllocationManager.java:54) ... 32 more ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Running the `spark-shell --remote` locally and `AmmoniteReplE2ESuite` on JDK 25 as: ``` ./build/mvn test -pl sql/connect/client/jvm -Dsuites="*.AmmoniteReplE2ESuite" ``` And also testing all the `ArrowConvertersSuite` tests: ``` $ build/sbt "sql/testOnly *ArrowConvertersSuite" ... [info] Run completed in 4 seconds, 719 milliseconds. [info] Total number of tests run: 43 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 43, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 127 s (0:02:07.0), completed May 20, 2026, 6:31:55 AM $ java --version openjdk 25.0.3 2026-04-21 LTS OpenJDK Runtime Environment Temurin-25.0.3+9 (build 25.0.3+9-LTS) OpenJDK 64-Bit Server VM Temurin-25.0.3+9 (build 25.0.3+9-LTS, mixed mode, sharing) ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #56006 from attilapiros/SPARK-56955. Authored-by: Attila Zsolt Piros <piros.attila.zsolt@gmail.com> Signed-off-by: attilapiros <piros.attila.zsolt@gmail.com> (cherry picked from commit dc1fde3) Signed-off-by: attilapiros <piros.attila.zsolt@gmail.com>

attilapiros · 2026-05-20T17:51:20Z

Merged to master, branch-4.x and branch-4.2.

Thanks @dongjoon-hyun, @pan3793, @peter-toth and @yadavay-amzn!

dongjoon-hyun · 2026-05-20T20:10:11Z

Thank you again for swift actions!

Initial upload.

9ec51c9

attilapiros mentioned this pull request May 20, 2026

[SPARK-56448][CONNECT] Fix NPE on Spark Connect client restart due to ammonite compile cache #55720

Closed

yadavay-amzn mentioned this pull request May 20, 2026

[SPARK-56955][CORE] Add --sun-misc-unsafe-memory-access=allow for JDK 25+ #55999

Closed

yadavay-amzn approved these changes May 20, 2026

View reviewed changes

Replacing all io.netty.noUnsafe=false with sun-misc-unsafe-memory-acc…

d2ba7bc

…ess=allow

pan3793 approved these changes May 20, 2026

View reviewed changes

peter-toth approved these changes May 20, 2026

View reviewed changes

dongjoon-hyun approved these changes May 20, 2026

View reviewed changes

dongjoon-hyun reviewed May 20, 2026

View reviewed changes

attilapiros changed the title ~~[SPARK-56955][CONNECT] Arrow/Netty memory allocator fails on JDK 25~~ [SPARK-56955][CONNECT] Replace -Dio.netty.noUnsafe=false with --sun-misc-unsafe-memory-access=allow for a more complete JDK 25+ support May 20, 2026

attilapiros closed this in dc1fde3 May 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-56955][CONNECT] Replace -Dio.netty.noUnsafe=false with --sun-misc-unsafe-memory-access=allow for a more complete JDK 25+ support#56006

[SPARK-56955][CONNECT] Replace -Dio.netty.noUnsafe=false with --sun-misc-unsafe-memory-access=allow for a more complete JDK 25+ support#56006
attilapiros wants to merge 2 commits into
apache:masterfrom
attilapiros:SPARK-56955

attilapiros commented May 20, 2026 •

edited

Loading

Uh oh!

yadavay-amzn left a comment •

edited

Loading

Uh oh!

pan3793 commented May 20, 2026 •

edited

Loading

Uh oh!

attilapiros commented May 20, 2026

Uh oh!

attilapiros commented May 20, 2026

Uh oh!

dongjoon-hyun left a comment

Uh oh!

dongjoon-hyun commented May 20, 2026

Uh oh!

dongjoon-hyun left a comment •

edited

Loading

Uh oh!

attilapiros commented May 20, 2026

Uh oh!

dongjoon-hyun commented May 20, 2026

Uh oh!

attilapiros commented May 20, 2026

Uh oh!

dongjoon-hyun commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

attilapiros commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

yadavay-amzn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

attilapiros commented May 20, 2026

Uh oh!

attilapiros commented May 20, 2026

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 20, 2026

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

attilapiros commented May 20, 2026

Uh oh!

dongjoon-hyun commented May 20, 2026

Uh oh!

attilapiros commented May 20, 2026

Uh oh!

dongjoon-hyun commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

attilapiros commented May 20, 2026 •

edited

Loading

yadavay-amzn left a comment •

edited

Loading

pan3793 commented May 20, 2026 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading