Conversation
github-actions
bot
added
CORE
|
Thanks for your hardwork, really appreciate it @WangGuangxin . I know Velox does certain queries where the result does not match with what vanilla spark outputs. Is this the same with Bolt? I think it would be nice to include some docs about any potential mismatches for users to be cautious about |
|
@WangGuangxin can you change the time metric of "total time of" to "time of" |
|
The initial performance number from our test, higher is better FYI. 
|
@afterincomparableyum thank you for your suggestion. We would add those documents later before this PR is merged. |
|
Can you create similar documents as Velox? here is the list of Velox docs. |
|
Does Bolt passes Spark3.2, 3.3, 3.4 and 3.5 UTs? We are going to drop 3.2 support and fixing 4.0 UTs. So Bolt may starts from 3.3 support. |
WangGuangxin
force-pushed
the
add_bolt_backend
branch
2 times, most recently
from
b871610 to
be4494f
Compare
Currently we mainly run UTs on spark3.5. For now most of 3.5 cound be passed, and we are fixing failed uts. We will follow the community's support for different spark versions. |
taiyang-li
force-pushed
the
add_bolt_backend
branch
from
283b3f4 to
7d328ca
Compare
* add make argument for specifing Bolt's version(branch/tag) * minor fix
WangGuangxin
force-pushed
the
add_bolt_backend
branch
from
39898e9 to
e74e874
Compare
* remove hardcoded java path * install jdk17 for centos dockerfile
|
Is there any documentation/script to build gluten jar with bolt backend? Like dev/buildbundle-veloxbe.sh ? I want to bench it against clickbench dataset with arm chip. |
@metegenez pls refer to steps in https://github.com/WangGuangxin/gluten/blob/d4ee706eb51a250f7bbacae70b46dffba62470b8/README.md |
@FelixYBW it had been fixed. Please get the lastest code and run: make bolt-recipe
make release ENABLE_S3=True |
|
iceberg table S3 read failed。hive table can be read successfully |
|
Velox PRs submited by Gluten committers and related: |
Iceberg read on s3 isn't enabled in Bolt yet. |
Currently to make sure llvm IR can call c/c++ code, we added a customized library loader which loads bolt backend with RTLD_GLOBAL flag. Since bolt_backend is a shared libraries, it will expose all the dynamic symbol table to Java process, and will cause symbols conflict if the user has other native libraries. Use version script to control the symbols exposure. How to check: ``` readelf -d -s --dyn-syms libbolt.so | grep jit_* ```
Signed-off-by: fangzhuhe <fangzhuhe@bytedance.com>
…egisterGcsFileSystem.h
…rc files to make sure Bolt can correctly handle legacy orc files
This change introduces all of the necessary paimon-specific changes that were included in olap/gluten's master branch but were not ported over when switching to OSS gluten's main branch The following changes were made - PaimonScanTransformer is now AbstractPaimonScanTransformer in order to support a other backend's implementation for paimon. Bolt-specific features are now included in an extended class BoltPaimonScanTransformer to facilitate the bolt-specific requirements. - Added support for adding extension info from the scan transformer into the advanced_extension field for a ReadRel. Also added protos for the paimon-specific advanced extension which are automatically generated and compiled for both Java and C++ versions. - The above also coincides with adding support for passing the "tableParameters" field of a HiveTableHandle when converting the substrait plan into bolt. - Added protos specific to paimon splits in order to serialize and deserialize paimon-specific split information from gluten into bolt. Previously this was done in a hacky way using strings and comma-separated lists. This new version uses protobuf both on the Java and C++ side to communicate paimon-specific split information via a LocalFiles' file_format field oneof definition in algebra.proto. One of the new options added is for "PaimonReadOptions" which contains the required paimon information. - Added a bolt-specific paimon suite which mirrors all of the previous test cases from the master branch.
Removed handling for StructType, ArrayType, and MapType in data type matching.
12 participants
No description provided.