Keep GPU FSST decompression on device

.github/workflows/pr-bench-gpu-compress.yml

@@ -32,6 +32,25 @@ jobs:
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           enable-sccache: "true"
+      - name: Install DuckDB
+        # The Public BI datasets are converted from CSV to Parquet through the DuckDB CLI, as
+        # in bench-pr.yml. Without it those datasets fail to materialise their fixture.
+        run: |
+          wget -qO- https://github.com/duckdb/duckdb/releases/download/v1.5.5/duckdb_cli-linux-amd64.zip | funzip > duckdb
+          chmod +x duckdb
+          echo "$PWD" >> "$GITHUB_PATH"
+      - name: Install uv
+        uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63  # 0.18.6
+      - name: Install cuDF
+        # The GPU Parquet number is a full cuDF `read_parquet`. cuDF ships prebuilt manylinux
+        # wheels on NVIDIA's index, so it stays a runtime dependency and never enters the Rust
+        # build. pandas and pyarrow back the --gpu-verify cross-check.
+        run: |
+          uv venv --python 3.12 .venv-cudf
+          uv pip install --python .venv-cudf \
+            --extra-index-url https://pypi.nvidia.com \
+            cudf-cu12 pandas pyarrow
+          echo "$PWD/.venv-cudf/bin" >> "$GITHUB_PATH"
       - uses: ./.github/actions/system-info
       - name: Display NVIDIA GPU details
         run: |
@@ -44,8 +63,68 @@ jobs:
           cargo build --locked --package compress-bench --profile release_debug --features cuda,unstable_encodings
       - name: Setup benchmark environment
         run: sudo bash scripts/setup-benchmark.sh
+      - name: Verify GPU decompression correctness
+        id: verify
+        shell: bash
+        continue-on-error: true
+        env:
+          RUST_BACKTRACE: "1"
+          FLAT_LAYOUT_INLINE_ARRAY_NODE: "1"
+        # Cross-checks every GPU-decompressed page and field against the CPU decoders before
+        # any timings are taken. Verification runs inline, so this pass is not timed. It runs
+        # every dataset rather than stopping at the first failure, so one run reports the whole
+        # matrix; the gate step below still fails the job.
+        run: |
+          set -o pipefail
+          target/release_debug/compress-bench \
+            --gpu-decompress --gpu-verify --iterations 1 -d table 2>&1 | tee gpu-verify.txt
+      - name: Publish verification result
+        shell: bash
+        # The per-dataset verdicts are worth surfacing on the PR whether or not they all pass:
+        # digging them out of a multi-thousand-line job log is otherwise the only way to see
+        # which encodings decode correctly on the GPU.
+        run: |
+          {
+            echo "# GPU decompression verification"
+            echo
+            if [ "${{ steps.verify.outcome }}" = "success" ]; then
+              echo "All GPU datasets matched the CPU decode."
+            else
+              echo "Verification failed. Per-dataset results:"
+            fi
+            echo
+            echo '```text'
+            grep -E "verified [0-9]+|GPU decompression failed|^  - " gpu-verify.txt | tail -40 \
+              || tail -40 gpu-verify.txt
+            echo '```'
+            # The per-dataset lines above carry only the first line of each error. Python
+            # tracebacks, Rust backtraces and first-differing-row dumps span several lines, so
+            # the tail of the raw output goes in a collapsed block rather than back in the log.
+            if [ "${{ steps.verify.outcome }}" != "success" ]; then
+              echo
+              echo "<details><summary>Full error detail</summary>"
+              echo
+              echo '```text'
+              tail -200 gpu-verify.txt
+              echo '```'
+              echo
+              echo "</details>"
+            fi
+          } > verify-comment.md
+          cat verify-comment.md >> "$GITHUB_STEP_SUMMARY"
+      - name: Comment PR with verification result
+        if: github.event.pull_request.head.repo.fork == false
+        uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b  # v3
+        with:
+          file-path: verify-comment.md
+          comment-tag: bench-pr-comment-gpu-verify
       - name: Run GPU compression benchmark
+        id: bench
         shell: bash
+        # A dataset the GPU cannot decode is reported per dataset and still fails the run, but
+        # the timing tables are printed first, so the datasets that do decode publish their
+        # numbers. The gate at the end of the job turns either failure into a job failure.
+        continue-on-error: true
         env:
           RUST_BACKTRACE: full
           # Do not enable VORTEX_EXPERIMENTAL_PATCHED_ARRAY here: it rewrites interior
@@ -56,9 +135,29 @@ jobs:
           # (cuda.yaml), which also sets only FLAT_LAYOUT_INLINE_ARRAY_NODE.
           FLAT_LAYOUT_INLINE_ARRAY_NODE: "1"
         run: |
+          # `shell: bash` implies -e, so a non-zero benchmark exit would skip the cat and leave
+          # the timing tables out of the job log entirely. Capture the status instead.
+          status=0
           bash scripts/bench-taskset.sh target/release_debug/compress-bench \
-            --gpu-decompress -d table > gpu-compress.txt
+            --gpu-decompress -d table > gpu-compress.txt || status=$?
           cat gpu-compress.txt
+          exit "$status"
+      - name: Run CPU decompression baseline
+        id: cpu-bench
+        continue-on-error: true
+        shell: bash
+        env:
+          RUST_BACKTRACE: full
+          FLAT_LAYOUT_INLINE_ARRAY_NODE: "1"
+        # Same binary, same datasets, same machine, CPU decoders. Without this the GPU ratio
+        # says which of two GPU readers is faster but not whether either beats the CPU, which
+        # is the only number that says whether GPU decoding is worth doing at all.
+        run: |
+          bash scripts/bench-taskset.sh target/release_debug/compress-bench \
+            --ops decompress \
+            --datasets 'TPC-H l_comment|taxi|Arade|Bimbo|CMSprovider|Euro2016|Food|HashTags' \
+            -d table > cpu-compress.txt
+          cat cpu-compress.txt
       - name: Publish results
         shell: bash
         run: |
@@ -68,6 +167,12 @@ jobs:
             echo '```text'
             cat gpu-compress.txt
             echo '```'
+            echo
+            echo "## CPU baseline (same machine, same datasets)"
+            echo
+            echo '```text'
+            cat cpu-compress.txt || echo "CPU baseline did not produce output"
+            echo '```'
           } > comment.md
           cat comment.md >> "$GITHUB_STEP_SUMMARY"
       - name: Comment PR
@@ -85,3 +190,11 @@ jobs:
 
             GPU Compression failed. Check the [workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details.
           comment-tag: bench-pr-comment-gpu-compress
+      - name: Fail if verification or the benchmark failed
+        if: steps.verify.outcome == 'failure' || steps.bench.outcome == 'failure'
+        shell: bash
+        # Last, so both the verification matrix and the timing tables are already published.
+        run: |
+          echo "verify=${{ steps.verify.outcome }} bench=${{ steps.bench.outcome }}" >&2
+          echo "GPU decompression failed; see the verification and results comments." >&2
+          exit 1

benchmarks/compress-bench/Cargo.toml

@@ -21,12 +21,15 @@ arrow-schema = { workspace = true }
 async-trait = { workspace = true }
 bytes = { workspace = true }
 clap = { workspace = true, features = ["derive"] }
+cudarc = { workspace = true, features = ["nvtx"], optional = true }
 futures = { workspace = true }
 indicatif = { workspace = true }
 itertools = { workspace = true }
 lance-bench = { path = "../lance-bench", optional = true }
 parquet = { workspace = true }
 regex = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
+serde_json = { workspace = true }
 tempfile = { workspace = true, optional = true }
 tokio = { workspace = true, features = ["full"] }
 tracing = { workspace = true }
@@ -36,7 +39,7 @@ vortex-bench = { workspace = true }
 vortex-cuda = { workspace = true, optional = true }
 
 [features]
-cuda = ["dep:tempfile", "dep:vortex-cuda"]
+cuda = ["dep:cudarc", "dep:tempfile", "dep:vortex-cuda"]
 lance = ["dep:lance-bench"]
 unstable_encodings = ["vortex/unstable_encodings", "vortex-cuda?/unstable_encodings"]
 
@@ -45,7 +48,6 @@ name = "compress-bench"
 test = false
 
 [lib]
-test = false
 
 [lints]
 workspace = true

benchmarks/compress-bench/README.md

@@ -15,13 +15,138 @@ See [`src/main.rs`](./src/main.rs) for the dataset list and CLI flags (`--format
 cargo run -p compress-bench --profile release_debug
 ```
 
-GPU decompression is opt-in and runs only the existing benchmark names allow-listed in
-`src/main.rs`:
+## GPU decompression
+
+`--gpu-decompress` is opt-in, requires the `cuda` feature, and restricts the suite to the
+GPU dataset list in `src/main.rs`. It measures decompression only, for two backends:
+
+- **Vortex** — the file is written with CUDA-compatible BtrBlocks encodings only
+  (`only_cuda_compatible`) and a CUDA flat layout, then decoded on the device all the way to
+  canonical arrays.
+- **Parquet** — the file is rewritten with GPU-friendly writer settings (see below) and read
+  back with [cuDF](https://github.com/rapidsai/cudf)'s `read_parquet`, which performs the
+  whole read on the device: page header decode, codec decompression, dictionary/RLE/plain
+  decoding and column assembly.
+
+Both sides therefore decode all the way to device-resident arrays, which is what makes the
+`vortex:parquet-<codec> gpu ratio decompress time` metric a like-for-like comparison.
+The generated files also use the same 1,048,576-row physical partition size: Parquet row groups
+and Vortex root chunks. Vortex input batches are concatenated and sliced at those exact boundaries
+before writing, so smaller source batches cannot leak into its on-disk layout.
 
 ```bash
 cargo run -p compress-bench --profile release_debug \
   --features cuda,unstable_encodings -- --gpu-decompress
+
+# pick the Parquet page codec the GPU file is written with (default: snappy)
+cargo run -p compress-bench --profile release_debug \
+  --features cuda,unstable_encodings -- --gpu-decompress --gpu-parquet-codec zstd
+
+# isolate one backend for diagnostics; the default remains parquet,vortex
+cargo run -p compress-bench --profile release_debug \
+  --features cuda,unstable_encodings -- --gpu-decompress --formats vortex
+```
+
+### Vortex GPU profiling
+
+`--gpu-vortex-profile wall|gpu|nsys` enables opt-in diagnostics for the Vortex backend. `wall`
+records host timings, `gpu` also brackets every field dispatch with CUDA events, and `nsys` adds
+per-field NVTX ranges. These modes perturb the measured run; use them to explain a result, then
+rerun without the flag for the comparison number.
+
+After each timed stream synchronization, the benchmark writes one JSON record to stderr with
+`record="vortex_gpu_decompress_profile"`. It includes file/layout sizes and counts, decoded rows,
+batches and field dispatches; `stages` contains microsecond wall times; and `encodings` groups calls,
+rows and wall time by full encoding tree and field name. In `gpu` mode, each encoding group also has
+`gpu_us`; it is `null` in the other modes.
+
+```bash
+cargo run -p compress-bench --profile release_debug \
+  --features cuda,unstable_encodings -- --gpu-decompress --formats vortex \
+  --datasets '^(Arade|Bimbo|CMSprovider)$' --iterations 3 \
+  --gpu-vortex-profile gpu 2> /tmp/vortex-gpu-profile.log
+
+# Ignore non-JSON progress/log lines and average the main stages by dataset.
+jq -Rs '
+  [split("\n")[] | fromjson? |
+   select(.record == "vortex_gpu_decompress_profile")]
+  | group_by(.dataset)
+  | map({
+      dataset: .[0].dataset,
+      runs: length,
+      total_us: (map(.stages.total_us) | add / length),
+      read_us: (map(.stages.read_us) | add / length),
+      dispatch_us: (map(.stages.field_dispatch_us) | add / length),
+      gpu_us: (map([.encodings[].gpu_us // empty] | add) | add / length)
+    })
+' /tmp/vortex-gpu-profile.log
+```
+
+`open_us` covers opening and footer metadata, `scan_plan_us` builds the scan stream, `read_us`
+awaits batches, `struct_dispatch_us` materializes each struct batch, `field_dispatch_us` measures
+CPU planning/enqueue time for field decodes, and `final_sync_us` is the remaining device tail.
+`profile_overhead_us` is the remainder spent collecting diagnostics, primarily encoding-tree
+formatting and CUDA-event bookkeeping; it makes the profiler's own perturbation explicit.
+Because CUDA work is asynchronous, `read_us` can include I/O, layout execution, backpressure, and
+waiting for earlier device work; it is not pure storage time. `gpu_us` is the device-stream time
+between field events. Allocation/free, upload, wait, event, and callback counts are not available
+from this record; use Nsight Systems for those runtime-wide counts.
+
+### cuDF
+
+cuDF is reached through its prebuilt `cudf-cu12` wheel, so it is a runtime dependency of the
+benchmark and never enters the Rust build:
+
+```bash
+uv pip install --extra-index-url https://pypi.nvidia.com cudf-cu12 pandas pyarrow
+```
+
+`scripts/cudf-parquet-read.py` performs and times the read. Timing is taken inside that
+script, so interpreter start, `import cudf` and CUDA context creation are excluded; a warm-up
+read runs first for the same reason.
+
+Both backends read a warm file by default. Each runs an untimed full read before a separately
+opened timed read, warming the OS page cache, allocator, and CUDA modules. Neither reuses decoded
+arrays, and the Vortex CUDA opener disables its data-segment cache. The Vortex reader therefore
+does **not** use direct I/O by default, because `O_DIRECT` would bypass the page cache and compare
+a Vortex read of the disk against a cuDF read of RAM. `--gpu-direct-io` turns it back on to measure
+storage bandwidth instead — a different question, and the resulting ratio is not a decode
+comparison.
+
+The remaining asymmetry is the transfer path: the Vortex reader uses pinned buffers, while cuDF
+does its own host read and host-to-device copy.
+
+### GPU-friendly Parquet writer settings
+
+Set in `src/gpu_writer.rs`:
+
+| Setting | Value | Why |
+| --- | --- | --- |
+| writer version | `PARQUET_1_0` | v1 pages compress the whole page body; v2 pages put uncompressed levels ahead of the compressed values in the same body. |
+| compression | Snappy (default) or Zstd | Snappy is the Parquet default and has the higher device-side throughput. |
+| dictionary | enabled | Keeps the decompressed payload small; the encoding GPU Parquet readers decode fastest. |
+| data page size | 1 MiB | Large enough to amortize per-page setup, small enough to keep every SM fed. Matches the page size cuDF targets. |
+| data page row limit | 1,000,000 | The 20k-row default caps narrow columns' pages far below 1 MiB. |
+| row-group / root-chunk rows | 1,048,576 | Gives both formats the same independently readable physical partitions and amortizes GPU launch overhead. |
+| statistics | chunk-level | Page statistics only inflate the headers a reader has to walk. |
+
+### Correctness
+
+`--gpu-verify` cross-checks device output against the CPU decoders on every iteration:
+
+- Parquet: the cuDF-read frame is compared against a CPU Parquet read of the same file.
+- Vortex: each GPU-decoded field is copied back and compared against the same field decoded
+  on the CPU, through Arrow with a pinned target type.
+
+Verification runs inline, so timings from a verifying run are not comparable to a plain one —
+run it as its own pass:
+
+```bash
+cargo run -p compress-bench --profile release_debug \
+  --features cuda,unstable_encodings -- --gpu-decompress --gpu-verify --iterations 1
 ```
 
-On Linux, GPU files are read with direct IO (`O_DIRECT`) so repeated iterations measure
-storage bandwidth rather than page-cache hits.
+Any `--gpu-decompress` run reports on every dataset rather than stopping at the first failure, so
+one run shows which datasets decode correctly on the GPU and which do not. The timing tables are
+rendered before the failure summary, so a dataset the GPU cannot decode still leaves the rest of
+the matrix with numbers — the process exits non-zero either way.