Compare Vortex GPU decompression against a cuDF Parquet read - #9147
Draft
joseph-isaacs wants to merge 17 commits into
Draft
Compare Vortex GPU decompression against a cuDF Parquet read#9147joseph-isaacs wants to merge 17 commits into
joseph-isaacs wants to merge 17 commits into
Conversation
The GPU compression benchmark only measured Vortex, and only on a single dataset, so it could not say anything about how Vortex GPU decompression compares to Parquet, nor about encodings beyond FSST strings. Parquet compresses each page body independently, which is exactly the batch shape nvCOMP's device decompressors take and how cuDF's Parquet reader gets pages off the CPU. This adds a Parquet backend built on that: column chunks are staged on the device through the same pinned, direct-I/O reader the Vortex backend uses, then every page in a row group is decompressed in one batched nvCOMP launch. - vortex-nvcomp: bind the batched Snappy decompression entrypoints and the per-algorithm alignment queries, and share `DecompressBackend` between the Snappy and Zstd wrappers. - compress-bench: locate compressed page bodies by walking the per-page Thrift headers (`parquet::format::PageHeader` is deprecated and `parquet`'s own parser is crate-private), and write files with GPU-friendly settings: v1 pages, dictionary encoding, 1 MiB pages, Snappy by default. - Run both Vortex and Parquet under `--gpu-decompress`, and expand the GPU dataset set from one to nine so ALP, bit-packed, run-end, date/time-parts and null-heavy columns are covered alongside FSST strings. - Add `--gpu-verify`, which compares every GPU-decompressed page against the host codec and every GPU-decoded Vortex field against the CPU decode, and run it as a CI step before the timed benchmark. Independently of that flag, nvCOMP's per-page status and size arrays are checked on every iteration. Page decoding is not part of the Parquet measurement, so its numbers are an upper bound on a full GPU Parquet reader; the README states this. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
A CUDA scan hands back arrays whose buffers live in device memory, so decoding those same arrays through the host Arrow path panics rather than producing a CPU reference. Read the file a second time through the ordinary host reader and compare the two scans batch by batch instead. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
Two changes to the Vortex GPU verification, after CI reported a `fastlanes.for` mismatch with no detail: - Read the CPU reference from a copy of the file. The session segment cache is keyed by URI and the CUDA reader deliberately bypasses it because its buffers are device-resident, so pointing both scans at one URI risks them sharing entries. - Synchronize the stream before copying a decoded field back, and report the Arrow types, lengths, null counts and the first differing row when the two decodes disagree. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
Merging this PR will degrade performance by 8.28%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | decompress[u64, (10000, 4)] |
310.2 µs | 401.7 µs | -22.77% |
| ❌ | Simulation | decompress[u64, (1000, 16)] |
64.4 µs | 72.8 µs | -11.57% |
| ❌ | Simulation | take[small_m/shuffled/primitive/nonnull/chunks=16384/indices=16] |
1 ms | 1.2 ms | -11.31% |
| ⚡ | Simulation | cold_misaligned[(64, 256)] |
5.1 ms | 4.4 ms | +16.82% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/gpu-decompress-benchmarks-4mmn93 (8e060ca) with develop (93b0535)
Footnotes
-
46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
The bit-unpack kernel writes patch values straight into the output while the lane decoder adds the frame of reference to every unpacked value. Bit-packing exceptions are stored in the same reference-relative domain as the packed values, so under `FoR(BitPacked)` every patched position came out short by exactly the reference. The existing kernel tests could not catch this: they exercise `BitPacked` directly, where the reference is zero. The new `FoRExecutor` case bit-packs to 8 bits with values that overflow into patches and a non-zero reference. Found by the compression benchmark's new `--gpu-verify` pass, which reported a `fastlanes.for` field decoding row 8038 as 131072 where the CPU produced 393061 — a difference of exactly the 261989 reference. Also thread the dataset name through compress-bench failures, so a benchmark error says which dataset it came from. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
A verification run stopped at the first dataset that failed, so finding the GPU-clean set took one CI cycle per dataset. Run every dataset instead, recording failures and reporting them together at the end, then exit non-zero. Missing CUDA kernel support surfaces as a panic rather than an error, so the survey catches those too. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
The per-dataset verification verdicts were only visible by digging through a multi-thousand-line job log. Capture the verification output, publish the per-dataset results to the step summary and a PR comment, and keep failing the job through a separate gate step. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
Contributor
GPU decompression verificationVerification failed. Per-dataset results: Full error detail |
The Public BI datasets build their Parquet fixture through the DuckDB CLI, as in bench-pr.yml. The GPU job never installed it, so all six failed with ENOENT before reaching the GPU at all. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
The nvCOMP backend only ran the codec stage on the device: page decoding stayed on the CPU and was excluded from the measurement, so the Parquet figure was an upper bound and the comparison against Vortex was not like-for-like. cuDF's `read_parquet` does the whole read on the device — page header decode, decompression, dictionary/RLE/plain decoding and column assembly — which is the same amount of work the Vortex backend does when it decodes to canonical arrays. It is reached through the prebuilt `cudf-cu12` wheel, so it stays a runtime dependency and never enters the Rust build. Timing is taken inside scripts/cudf-parquet-read.py, so interpreter start, `import cudf` and CUDA context creation are excluded; a warm-up read runs first. `--gpu-verify` now compares the cuDF frame against a CPU Parquet read. This removes the page scanner, the batched nvCOMP launch path and the nvCOMP Snappy bindings, all of which existed only to serve the codec-stage backend. What remains of the Parquet side is the GPU-friendly writer settings, now in gpu_writer.rs. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
force-pushed
the
claude/gpu-decompress-benchmarks-4mmn93
branch
from
August 13, 2026 15:04
c93d5cc to
64d481c
Compare
joseph-isaacs
added
action/benchmark-gpu-compress
and removed
action/benchmark-gpu-compress
labels
Claude
The reference side of the Vortex verification was executing through the CUDA context: the host scan's batches and both Arrow conversions were handed `cuda_ctx.execution_ctx()`. A CUDA context allocates its outputs in device memory, so the Arrow conversion then read a device buffer from the host and panicked with "unwrap_host called for Device allocation" on the string-heavy Public BI datasets, where canonicalisation goes through the buffer directly. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
…ess-benchmarks-4mmn93 Signed-off-by: Claude <noreply@anthropic.com>
The verification gate ran before the timed pass, so a single unsupported dataset meant the run produced no numbers at all. Two changes: - the per-dataset survey now applies to any GPU run, not just a verifying one, and the timing tables render before the failure summary, so datasets that do decode still publish their numbers; - the workflow runs the benchmark before the gate and fails the job at the end on either a failed verification or a failed benchmark. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
changed the title
joseph-isaacs
added
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
Signed-off-by: Claude <noreply@anthropic.com>
The per-dataset grep captures only the first line of each error, so a Python traceback from the cuDF script or a Rust backtrace never reached the comment. Attach the tail of the raw output in a collapsed block on failure, and match the summary line's current wording. Signed-off-by: Claude <noreply@anthropic.com>
fix(cuda): copy validity back to the host in `into_host` `CanonicalCudaExt::into_host` migrated a canonical array's values buffer but passed its validity through untouched, so a nullable array came back to the host half-migrated and the first host read of the mask panicked with "unwrap_host called for Device allocation" — via `Validity::execute_mask` -> `BoolArray::into_bit_buffer`. Non-nullable arrays were unaffected, which is why it only showed up on the Public BI tables. The `Bool` arm already carried a TODO for exactly this. Do not compare Parquet DATE columns across representations pyarrow materialises a DATE column as `datetime.date` objects and cuDF as `datetime64[s]`. The values agree, but `check_dtype=False` does not bridge object-vs-datetime64, so the comparison reported every row as different and failed both TPC-H datasets. Coerce both sides to datetime64 first. Read the Vortex GPU file through the page cache by default cuDF takes an untimed warm-up read, so its timed read is served from the page cache, while the Vortex reader used `O_DIRECT` on every iteration and paid real disk reads each time. That compared a read of the disk against a read of RAM. Direct IO is now off by default and available behind `--gpu-direct-io` for measuring storage bandwidth, which is not a decode comparison. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
The GPU ratio says which of two GPU readers is faster, not whether either beats the CPU decoders. Run the same binary over the same datasets with the CPU path on the same machine and publish it alongside, so the GPU numbers can be read against something. Also capture the benchmark's exit status rather than letting `shell: bash`'s -e skip the `cat`, which kept the timing tables out of the job log and left them only in the PR comment. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
The workflow had grown a verification pass, a CPU baseline pass, two extra PR comments and a separate failure gate. None of that is needed to report a Parquet and a Vortex number. Reset the file to its develop version and add back only what the benchmark cannot run without: - the DuckDB CLI, which builds the Public BI Parquet fixtures; - uv and the cuDF wheel, which the GPU Parquet backend shells out to. Every pre-existing step is now untouched, and `--gpu-verify` remains available as a local flag. Signed-off-by: Claude <noreply@anthropic.com>
Parquet was read as ~1M-row row groups while Vortex inherited the Arrow reader's ~8K-row batches, so the Vortex file carried hundreds of small chunks where Parquet carried a handful of row groups. A chunk is the unit the reader plans and dispatches over, so that turned single dispatches into hundreds of small kernel launches and made the two numbers measure different shapes of work rather than two decoders. Pin both to `GPU_ROW_GROUP_SIZE` (1,048,576 rows, Parquet's `DEFAULT_MAX_ROW_GROUP_ROW_COUNT`): - state the Parquet row group count explicitly so it moves with the constant; - add `parquet_to_vortex_chunks_with_batch_size`, which concatenates the source batches and re-slices on exact boundaries. Setting the Arrow reader's batch size alone is not enough, because the reader also breaks at the source file's row group boundaries and still emits short batches; - write those batches through as root chunks with `ChunkedLayoutStrategy` and read them back with `SplitBy::RowCount(GPU_ROW_GROUP_SIZE)`. Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added
the
action/bench-gpu-compress
Claude
github-actions
Bot
removed
the
action/bench-gpu-compress
Contributor
BENCHMARK FAILEDGPU Compression failed. Check the workflow run for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The GPU decompression benchmark measured only Vortex, so the
--gpu-decompressnumbers hadnothing to compare against — a device-side decode rate is only meaningful next to another
format decoded on the same device.
The comparison point is cuDF's
read_parquet, which doesthe whole Parquet read on the GPU: page header decode, codec decompression, dictionary/RLE/plain
decoding and column assembly. Both sides therefore decode all the way to device-resident arrays,
which is what makes the ratio a like-for-like number rather than a comparison against a host read.
Adding a second decoder also gave the Vortex CUDA path a reference to be checked against, and
that cross-check found two real correctness bugs in
vortex-cuda(see below).What changes are included in this PR?
GPU Parquet backend.
benchmarks/compress-bench/src/gpu_parquet.rsrewrites each datasetwith GPU-friendly writer settings and times a cuDF read of it. cuDF is reached through its
prebuilt
cudf-cu12manylinux wheel and driven byscripts/cudf-parquet-read.py, so it is aruntime dependency of the benchmark and never enters the Rust build. Timing is taken inside the
script after a warm-up read, so interpreter start,
import cudfand CUDA context creation areexcluded.
GPU-friendly Parquet writer settings (
src/gpu_writer.rs): v1 pages, Snappy (default) orZstd, dictionary enabled, 1 MiB data pages with a 1,000,000-row limit, chunk-level statistics.
The rationale for each is tabulated in the README.
Like-for-like I/O. cuDF takes an untimed warm-up read, so its timed read is served from the
page cache. The Vortex reader therefore no longer uses
O_DIRECTby default — doing so compareda Vortex read of the disk against a cuDF read of RAM on every iteration.
--gpu-direct-iorestores it for measuring storage bandwidth, which is a different question and not a decode
comparison.
Correctness.
--gpu-verifycross-checks both backends against the CPU decoders inline: thecuDF frame against a CPU Parquet read, and each GPU-decoded Vortex field against a separate
host-only scan of a copy of the file, compared through Arrow with a pinned target type. A
mismatch reports the differing types, lengths and null counts plus the first differing row,
found by binary search.
Two
vortex-cudabugs found by the cross-check(exception) values without adding the reference, and patches are stored reference-relative —
so any patched value under
FoR(BitPacked)decoded on the GPU came back short by exactly thereference. Fixed in
vortex-cuda/src/bit_unpack_gen.rs, kernels regenerated, with an rstestregression case in
vortex-cuda/src/kernel/encodings/for_.rscovering u32/u64 patches atlane, block and cross-block boundaries.
into_hostleft validity on the device.CanonicalCudaExt::into_hostmigrated acanonical array's values buffer but passed its
Validitythrough untouched, so a nullablearray came back half-migrated and the first host read of the mask panicked in
BufferHandle::unwrap_host(viaValidity::execute_mask→BoolArray::into_bit_buffer).Non-nullable arrays were unaffected, which is why it only surfaced on the nullable Public BI
tables. The
Boolarm already carried a TODO for exactly this.Benchmark coverage.
--gpu-decompressnow runs nine datasets (TPC-Hl_commentcanonicaland chunked, taxi, and the Arade/Bimbo/CMSprovider/Euro2016/Food/HashTags Public BI tables)
rather than one, and reports on every dataset instead of stopping at the first failure — one run
shows the whole matrix. The timing tables render before the failure summary, so datasets that do
decode still publish numbers when another dataset cannot.
CI.
pr-bench-gpu-compress.ymlinstalls the DuckDB CLI (needed to build the Public BIfixtures, as in
pr-bench-compress.yml) and the cuDF wheel, runs a verification pass and thetimed pass, publishes both to the PR — attaching full tracebacks and backtraces on failure —
and fails the job at the end if either failed.
Known gaps
Two CUDA encoding gaps remain, both in
vortex-cudarather than in the benchmark, and bothoutside the scope of this PR:
taxiandArade:Unsupported ptype u16. The CUDAdate_time_partskernel dispatches withmatch_each_signed_integer_ptype!while the CPU canonicaliser usesmatch_each_integer_ptype!.Widening the fused kernel's dispatch turns 4³ = 64 PTX instantiations into 8³ = 512, so the fix
is not free.
Euro2016andHashTags:No CUDA kernel for encoding vortex.masked.What APIs are changed? Are there any user-facing changes?
One library behaviour change:
CanonicalCudaExt::into_hostnow migrates validity as well asvalues, so a nullable canonical array copied back from the device is fully host-resident. That
is a bug fix — the previous result panicked on first use.
Everything else is confined to the
compress-benchbinary. New benchmark CLI flags:--gpu-parquet-codec,--gpu-verifyand--gpu-direct-io. Running--gpu-decompressnowadditionally requires the
cudf-cu12wheel onPATH;benchmarks/compress-bench/README.mddocuments the install and the remaining transfer-path asymmetry (the Vortex reader uses pinned
buffers; cuDF does its own host read and host-to-device copy).