Implement compare/between for bitpacked arrays - improve performance by up to 2X#7279
Implement compare/between for bitpacked arrays - improve performance by up to 2X#7279
Conversation
Merging this PR will improve performance by ×2.5
Performance Changes
Comparing Footnotes
|
74732ef to
8cddde0
Compare
Polar Signals Profiling ResultsLatest Run
Previous Runs (4)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals ProfilingVortex (geomean): 0.992x ➖ datafusion / vortex-file-compressed (0.992x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (1.025x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.022x ➖, 0↑ 0↓)
datafusion / parquet (1.024x ➖, 2↑ 2↓)
datafusion / arrow (1.014x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed (1.028x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.027x ➖, 0↑ 1↓)
duckdb / parquet (0.993x ➖, 3↑ 1↓)
duckdb / duckdb (1.013x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.917x ➖, 4↑ 0↓)
datafusion / vortex-compact (0.994x ➖, 0↑ 1↓)
datafusion / parquet (0.970x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.046x ➖, 0↑ 1↓)
duckdb / vortex-compact (0.992x ➖, 1↑ 0↓)
duckdb / parquet (1.009x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.958x ➖, 9↑ 0↓)
datafusion / vortex-compact (0.950x ➖, 4↑ 0↓)
datafusion / parquet (0.969x ➖, 10↑ 2↓)
duckdb / vortex-file-compressed (0.936x ➖, 18↑ 0↓)
duckdb / vortex-compact (0.939x ➖, 15↑ 0↓)
duckdb / parquet (0.955x ➖, 8↑ 1↓)
duckdb / duckdb (0.944x ➖, 13↑ 0↓)
Full attributed analysis
|
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (1.046x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.046x ➖, 0↑ 0↓)
datafusion / parquet (1.037x ➖, 0↑ 0↓)
datafusion / arrow (1.053x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.031x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.043x ➖, 0↑ 0↓)
duckdb / parquet (1.019x ➖, 0↑ 1↓)
duckdb / duckdb (1.022x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) duckdb / vortex-file-compressed (0.948x ➖, 1↑ 0↓)
duckdb / vortex-compact (0.987x ➖, 0↑ 0↓)
duckdb / parquet (0.988x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: Clickbench on NVMEVerdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (1.298x ❌, 0↑ 42↓)
datafusion / parquet (1.247x ❌, 0↑ 41↓)
duckdb / vortex-file-compressed (1.239x ❌, 1↑ 42↓)
duckdb / parquet (1.166x ❌, 0↑ 39↓)
duckdb / duckdb (1.094x ➖, 0↑ 16↓)
Full attributed analysis
|
Benchmarks: FineWeb S3Verdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.997x ➖, 1↑ 1↓)
datafusion / vortex-compact (0.925x ➖, 0↑ 0↓)
datafusion / parquet (1.071x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.032x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.021x ➖, 0↑ 0↓)
duckdb / parquet (0.990x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: TPC-H SF=1 on S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (1.092x ➖, 0↑ 3↓)
datafusion / vortex-compact (0.965x ➖, 2↑ 0↓)
datafusion / parquet (1.026x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.029x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.953x ➖, 1↑ 0↓)
duckdb / parquet (0.965x ➖, 1↑ 0↓)
Full attributed analysis
|
Benchmarks: TPC-H SF=10 on S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (0.920x ➖, 3↑ 0↓)
datafusion / vortex-compact (0.935x ➖, 2↑ 1↓)
datafusion / parquet (1.106x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed (1.059x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.083x ➖, 0↑ 0↓)
duckdb / parquet (0.963x ➖, 0↑ 0↓)
Full attributed analysis
|
a103738 to
4454bc6
Compare
52289bf to
dbe5a37
Compare
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
dbe5a37 to
33e6d30
Compare
After some iterations, this seems to improve performance in any case we actually measure. It does require merging and releasing spiraldb/fastlanes#125.