Skip to content

bench: run the full compress suite under --gpu-decompress - #9399

Draft
joseph-isaacs wants to merge 11 commits into
developfrom
claude/gpu-compress-benchmarks-x788xc
Draft

bench: run the full compress suite under --gpu-decompress#9399
joseph-isaacs wants to merge 11 commits into
developfrom
claude/gpu-compress-benchmarks-x788xc

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

--gpu-decompress in compress-bench was gated behind an incremental allow-list holding a
single entry (TPC-H l_comment canonical), so the GPU compression benchmark only ever measured
one dataset. Widening it to the whole suite surfaced six independent gaps in the GPU decode path,
each fixed here. The benchmark now runs all 17 datasets end to end on the GPU.

What changes are included in this PR?

Benchmark scope

  • Remove the gpu_decompress_benchmarks allow-list. --gpu-decompress now runs every dataset in
    the suite: taxi, the Public BI sets, both TPC-H l_comment variants, the StructListOfInts
    wide-table variants, plus airquality and rplace.
  • Install the DuckDB CLI in the GPU benchmark workflow. The Public BI CSV-to-parquet conversion
    shells out to it, and the CPU benchmark runner already did this.

GPU decode gaps found by the widened suite

  • DateTimeParts dispatched through match_each_signed_integer_ptype!, panicking with
    "Unsupported ptype u16" on taxi. Compression picks the narrowest ptype per component and the
    CPU decoder already accepts any integer ptype, so match that and generate the kernel for every
    signed and unsigned width.
  • RunEnd bailed when values carried Validity::Array. The error claimed a CPU fallback, but
    execute_cuda can only fall back while all buffers are host-resident, so a device-resident scan
    failed outright. New runend_bool kernel expands the per-run validity bitmap through the run
    mapping, one output byte per thread.
  • No CUDA kernel for vortex.masked. New MaskedExecutor: a MaskedArray is a child carrying no
    nulls plus the bitmap that supplies them, so decode both on the GPU and attach the mask.
  • No CUDA kernel for vortex.list. New ListExecutor and list_views kernel deriving the
    ListView offset/size pair from List's len + 1 Arrow offsets.
  • only_cuda_compatible() excluded the integer and float sparse schemes but missed
    string::NullDominatedSparseScheme, so the CUDA-compatible compressor still emitted
    vortex.sparse (which has no GPU decoder) for string columns.
  • into_host copied data buffers off the device but left Validity::Array bitmaps on it
    ("unwrap_host called for Device allocation"), and had no Canonical::List arm at all.

Tests

  • New coverage for unsigned DateTimeParts components, RunEnd validity expansion across byte
    boundaries, masked validity, and list decode (contiguous, empty, and nullable).
  • The nullable RunEnd tests now call the executor directly. Going through execute_cuda silently
    falls back to CPU for a host-resident array, so they passed without ever running the GPU kernel.

What APIs are changed? Are there any user-facing changes?

No public library API changes. compress-bench --gpu-decompress no longer filters the dataset
list, and three encodings (vortex.masked, vortex.list, and RunEnd with per-element validity)
now decode on the GPU instead of erroring.

Checks

  • cargo clippy -p vortex-cuda -p vortex-btrblocks -p compress-bench --all-targets, cargo +nightly fmt --all, cargo test -p vortex-btrblocks --lib builder, and
    yamllint --strict all pass locally.
  • CI CUDA tests (plus memcheck, initcheck, synccheck) pass on real hardware.
  • The GPU compression benchmark completes all 17 datasets.

One thing worth a reviewer's judgement: the GPU path now downloads airquality and rplace from
pcodec's public S3 bucket on every run. The CPU-side exclusion of those two exists to avoid
creating egress charges for pcodec, so this may want caching rather than a standing CI cost.

claude added 2 commits August 13, 2026 12:17
Drop the incremental GPU allow-list so --gpu-decompress runs every dataset
in the compress suite, and include the airquality dataset on the GPU path.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
airquality and rplace come from the same pcodec bucket, so gate them
consistently: the GPU suite runs every compress dataset.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
@codspeed-hq

codspeed-hq Bot commented Aug 13, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 1.6%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 2035 untouched benchmarks
⏩ 46 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime words_gather_scalar[65536] 8.3 µs 9.4 µs -11.98%
WallTime words_gather_dispatch[1024] 33 ns 30 ns +10%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/gpu-compress-benchmarks-x788xc (8d6595e) with develop (1a493b6)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

GPU Compression

┌───────────────────────────────────────────────────────────┬────────────────────────┐
│ Benchmark                                                 │ vortex-file-compressed │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/taxi                                      │ 109761.00 μs (1.00)    │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/Arade                                     │ 548279.00 μs (1.00)    │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/Bimbo                                     │ 1428127.00 μs (1.00)   │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/CMSprovider                               │ 6595018.00 μs (1.00)   │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/Euro2016                                  │ 928742.00 μs (1.00)    │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/Food                                      │ 354424.00 μs (1.00)    │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/HashTags                                  │ 2120291.00 μs (1.00)   │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/TPC-H l_comment chunked                   │ 1963364.00 μs (1.00)   │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/TPC-H l_comment canonical                 │ 1956508.00 μs (1.00)   │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/rplace                                    │ 2597239.00 μs (1.00)   │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/airquality                                │ 21991.00 μs (1.00)     │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/wide table cols=100 chunks=1 rows=1000    │ 7089.00 μs (1.00)      │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/wide table cols=1000 chunks=1 rows=1000   │ 59950.00 μs (1.00)     │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/wide table cols=10000 chunks=1 rows=1000  │ 645809.00 μs (1.00)    │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/wide table cols=100 chunks=50 rows=1000   │ 7019.00 μs (1.00)      │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/wide table cols=1000 chunks=50 rows=1000  │ 61531.00 μs (1.00)     │
├───────────────────────────────────────────────────────────┼────────────────────────┤
│ decompress time/wide table cols=10000 chunks=50 rows=1000 │ 639750.00 μs (1.00)    │
└───────────────────────────────────────────────────────────┴────────────────────────┘

The CUDA DateTimeParts executor dispatched with match_each_signed_integer_ptype,
so it panicked with "Unsupported ptype u16" on the taxi benchmark. Compression
picks the narrowest ptype per component, and the CPU decoder already accepts any
integer ptype, so match the CPU behaviour and generate the kernel for every
signed and unsigned integer width.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
@joseph-isaacs joseph-isaacs added the changelog/fix A bug fix label Aug 13, 2026 — with Claude
claude added 2 commits August 13, 2026 17:01
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RunEnd GPU decoding bailed when values carried Validity::Array. The error
message claimed a CPU fallback, but execute_cuda can only fall back while all
buffers are host-resident, so a device-resident scan (as in the GPU compression
benchmark) failed outright.

Add a runend_bool kernel that expands the per-run validity bitmap through the
same run mapping as the values. Each thread owns a whole output byte so threads
never race on bits within a byte.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
The Public BI datasets are converted from CSV with the DuckDB CLI, which the
CPU benchmark runner already installs. The GPU job never needed it while its
suite was a single TPC-H dataset; now that it runs the full suite, Arade fails
with "No such file or directory" when spawning duckdb.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
The GPU compression benchmark failed on Euro2016 with "No CUDA kernel for
encoding vortex.masked". A MaskedArray is a child array that carries no nulls
of its own plus the validity bitmap that supplies them, so decode the child on
the GPU, decode the mask on the GPU, and attach the mask to the result.

Bail when the child itself carries a per-element validity bitmap: intersecting
two device-resident bitmaps would need a CPU compute pass, and MaskedArray's
own invariant makes that case unreachable for well-formed arrays.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
claude added 2 commits August 13, 2026 17:47
Canonical arrays keep validity as a separate child array, and into_host moved
only the data buffer, leaving a device-resident bitmap that CPU compute cannot
read ("unwrap_host called for Device allocation"). Any executor returning
Validity::Array from the GPU hit this; the MaskedExecutor test caught it.

Also call the executors directly in the nullable RunEnd tests. Going through
execute_cuda silently falls back to CPU for a host-resident array, so those
tests passed without ever running the new runend_bool kernel.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tible

vortex.sparse has no CUDA decode kernel, and only_cuda_compatible already
excluded the integer and float sparse schemes. The string variant was missed,
so Euro2016 failed the GPU compression benchmark with "No CUDA kernel for
encoding vortex.sparse".

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 13, 2026
The GPU compression benchmark failed on the StructListOfInts datasets with
"No CUDA kernel for encoding vortex.list". The CUDA side already had ListView
offset kernels and an Arrow device export path, but nothing produced a
device-resident ListView from the compressed List encoding.

List stores len + 1 Arrow-style offsets; its canonical form stores one offset
and one size per list. Decode the elements on the GPU and derive the view pair
from the offsets with a single kernel, writing both outputs from one thread.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 14, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 14, 2026
into_host had no Canonical::List arm, so it hit its todo! fallback with "list(i32)
not implemented" as soon as a ListViewArray was produced on the GPU. Copy the
elements, offsets, and sizes children back to the host alongside the validity
bitmap.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs added the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 14, 2026 — with Claude
@github-actions github-actions Bot removed the action/bench-gpu-compress Run only the GPU compression benchmark on this PR label Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants