Introduce writeBatchFfi in Java JNI to avoid IPC serialization overhead #7204
JingsongLi
started this conversation in
Feature Requests
Replies: 1 comment
-
|
Tagging @robert3005 as he's revamping the Java bindings at the moment |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Currently writeBatch accepts Arrow IPC serialized bytes (byte[]), which requires serialization/deserialization overhead. We wants a writeBatchFfi path that directly accepts raw Arrow C Data Interface pointers (arrow_array_addr and arrow_schema_addr), avoiding the IPC round-trip.
We have conducted some internal tests, and the cost of serialization may reach 20% of the cost of writing.
(From Apache Paimom community)
Beta Was this translation helpful? Give feedback.
All reactions