Skip to content

Add Scala Native support with ported Jox channels#445

Open
adamw wants to merge 17 commits into
masterfrom
scala-native-support
Open

Add Scala Native support with ported Jox channels#445
adamw wants to merge 17 commits into
masterfrom
scala-native-support

Conversation

@adamw
Copy link
Copy Markdown
Member

@adamw adamw commented May 25, 2026

Summary

  • Add Scala Native 0.5.12 cross-compilation support to the core module
  • Port the Jox channels library (lock-free segmented channel with select) from Java to Scala, using AtomicLong/AtomicReference/AtomicReferenceArray in place of VarHandle (not available in Scala Native)
  • Native channel wrapper delegates to the Scala port in the same pattern as the JVM wrapper delegates to Java Jox

Structure

  • core/shared/ — all platform-independent Ox code (structured concurrency, flows, resilience, scheduling)
  • core/jvm/ — JVM wrapper delegating to com.softwaremill.jox (unchanged behavior)
  • core/native/ — Scala Jox port (ox.channels.jox package) + native wrapper + tests

Jox port details

Ported from jox 1.1.2 (v1.1.2-channels tag). Divergences documented in core/native/src/main/scala/ox/channels/jox/README.md:

  1. AtomicXxx instead of VarHandle (Scala Native stubs VarHandle but doesn't implement it)
  2. AtomicReference[Segment] parameter instead of VarHandle + owner object in findAndMoveForward
  3. No forEach/toList on Source (provided by Ox wrapper)
  4. No Sink.trySend(value, channels...) (provided by Ox select API)
  5. Simplified toString

Test plan

  • All 899 JVM tests pass (no regressions)
  • 51 native tests pass (rendezvous, buffered, unlimited, closed, trySend/tryReceive, interruption, select)
  • Native test compilation and linking verified with clang 19
  • CI with Scala Native toolchain (requires clang in CI environment)

🤖 Generated with Claude Code

adamw and others added 4 commits May 24, 2026 16:02
@adamw @claude
Add Scala Native support with ported Jox channels implementation
8d7e0d2 
- Add sbt-scala-native and sbt-crossproject plugins
- Convert core module to crossProject(JVM, Native) with Full cross type
- Move shared source to core/shared/, JVM-specific to core/jvm/, Native to core/native/
- Port the Jox channels library (Channel, Segment, Select, Continuation) from Java to Scala
  in ox.channels.jox package, using AtomicReference/AtomicLong/AtomicInteger instead of VarHandle
- Native channel wrapper delegates to the Scala Jox port (same pattern as JVM delegates to Java Jox)
- Enable multithreading in native config (requires Scala Native 0.5.12 with virtual thread support)
- All 923 JVM tests pass; native compilation succeeds

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Document Jox port divergences, versions, and removal conditions
f0009f6 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Fix critical doSend bug, eliminate recursive retries, move jox port t…
085b1cb 
…o native-only

Fixes:
- doSend: FAILED result in segment-forward-resolution branch now correctly
  retries instead of returning null (success). Bug was caused by `return`
  before a match expression.
- trySendOrClosed/tryReceiveOrClosed: replaced recursive retry with loop
  via RETRY_SENTINEL to prevent stack overflow under extreme contention.

Structural:
- Move ox.channels.jox package from shared/ to native/ (no dead code on JVM)
- Expand test coverage: port ChannelClosedTest, ChannelInterruptionTest,
  more SelectTest cases, more TrySendReceive tests from Java jox 1.1.2
- Add TestUtil.scala (scoped/fork/forkCancelable) matching Java TestUtil
- Document test mapping in README (what's ported, what's not and why)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Fix native test execution: use %%% for ScalaTest, exclude shared test…
358c034 
…s from native link

- Use ScalaTest %%% (cross-platform) dependency for core cross-project
- Set Test/fork := false for native (required by Scala Native test runner)
- Exclude shared test sources from native (they use JVM-only APIs: java.time, java.net)
- Reduce many-forks test count from 1000 to 100 for native scheduling reliability
- All 51 native tests pass; all 899 JVM tests pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@He-Pin
Copy link
Copy Markdown

He-Pin commented May 25, 2026

AtomicXxx instead of VarHandle (Scala Native stubs VarHandle but doesn't implement it)
Is it possible to submit a pr to scala native,and implement VarHandle in scala native?

adamw and others added 9 commits May 25, 2026 10:51
@adamw @claude
Add virtual threads benchmark example (JVM vs Native)
d5d76fb 
Spawns 10,000 virtual threads in an Ox supervised scope, measures
wall-clock time. Includes README with instructions for packaging
and running standalone binaries on both platforms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Rename example→examples, bump to 100k threads, fix paths, add fat-jar…
8f7f0c6 
… packaging

- Rename to VirtualThreadsNativeJvmBenchmark, module to examples
- Bump from 10k to 100k virtual threads
- Add sbt-assembly for fat-jar packaging
- Move README content into class scaladoc with verified paths
- Retain only fat-jar (JVM) and native-binary methods

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw
Format code
77fdb6f
@adamw @claude
Apply scalafmt to native test sources
09d4760 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Add source back-pointers to all ported Jox files
0f2d100 
Each file now has a comment linking to the original Java source in
the softwaremill/jox repo at the v1.1.2-channels tag.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Migrate from sbt-crossproject to sbt-projectmatrix
e7dff70 
Replace sbt-scala-native-crossproject with sbt-projectmatrix (0.11.0),
matching the convention used in other SoftwareMill projects (sttp, sttp-model).

Source layout changes:
- core/shared/src/ → core/src/main/scala/, core/src/test/scala/
- core/jvm/src/ → core/src/main/scalajvm/, core/src/test/scalajvm/
- core/native/src/ → core/src/main/scalanative/, core/src/test/scalanative/

Project names: core3 (JVM), coreNative3 (Native), examples3, examplesNative3.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw
Format code
0ff933b
@adamw
Update example
739a47e
@adamw
Update docs
0aa5d36
@adamw
Copy link
Copy Markdown
Member Author

adamw commented May 25, 2026

@He-Pin as for VarHandle availability, maybe @WojciechMazur has some insights, as it's all his doing :)

@WojciechMazur
Copy link
Copy Markdown

WojciechMazur commented May 25, 2026 •
edited
Loading

Regarding VarHandles - to be able to support them we'd need to introduce special handling in the compiler plugin - we don't have real reflection, so all values need to be resolved and preserved at compile time. It's doable, but was always delayed. In SN to replace VarHandles we typically use a combination of scalanative.runtime.Intrinsics.classFieldRawPtr + C stdlib atomics (example here, and another example for AtomicFieldUpdater) - if you want to reduce allocations that might be the best way go without waiting for changes in the upstream
I can try to work on that for 0.5.13 when I find some spare time, but still most of the differences would be around the some additional allocation + indirection. Eventually it might be useful, but for first iteration j.u.c.atomic might be just fine

adamw and others added 4 commits May 25, 2026 20:45
@adamw @claude
Run shared tests on Native: fix Trail.scala, move IO tests to scalajvm
181e9ec 
- Replace java.time.Clock in Trail.scala with System.currentTimeMillis()
  (Clock not available on Scala Native)
- Move FlowCompanionIOOpsTest and FlowIOOpsTest to scalajvm/ (use java.net.URL,
  Class.getResource not available on Native)
- Remove native test source restriction — all shared tests now compile and
  link on Native (~873 tests run successfully, 11 failures from timing/select issues)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Fix Trail to use java.util.Date, use Integer.valueOf(0) in Jox port
6a559ce 
- Trail.scala: use java.util.Date for human-readable timestamps
  (java.time.Clock not available on Scala Native)
- Channel.scala: replace 0.asInstanceOf[AnyRef] with Integer.valueOf(0)
  to ensure non-null boxed value on all platforms

Investigation: ActorTest hangs on Native due to Scala Native 0.5.12
virtual thread scalability limitation — 1000+ virtual threads
concurrently blocking on CompletableFuture.get() causes starvation.
Works at N<=100. Not a Jox port bug.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Add repro for Scala Native virtual thread scalability issue
3d2701a 
NativeVirtualThreadScalabilityIssue.scala demonstrates that N>=500
virtual threads concurrently blocking on CompletableFuture.get()
causes a livelock on Scala Native 0.5.12 (works on JVM at any N).

This explains the ActorTest hang on Native.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw @claude
Simplify virtual thread scalability repro
71bb9d9 
Use Thread.join() instead of CompletableFuture for fork tracking,
removing one layer of indirection.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adamw
Copy link
Copy Markdown
Member Author

adamw commented May 27, 2026

@WojciechMazur great, yeah definitely using atomics is fine for the first pass

@adamw
Copy link
Copy Markdown
Member Author

adamw commented May 27, 2026

Unfortunately there seems to be a problem with scala-native's virtual thread scheduling (as demonstrated by NativeVirtualThreadScalabilityIssue.scala), so we'll have to wait with merging until this is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adamw @He-Pin @WojciechMazur