Add deletion vector loader by jihoonson · Pull Request #14294 · NVIDIA/spark-rapids

jihoonson · 2026-02-18T23:18:48Z

Description

This PR adds a deletion vector loader for Delta scan that loads serialized deletion vectors (roaring bitmaps) into host memory. As Delta serialization formats are different from the standard serialization format described in #14263, the loader performs the format conversion and returns a HostMemoryBuffer containing the bitmap serialized in the standard format.

Note that many pieces of the code are adopted from Delta classes such as DeletionVectorStore and StoredBitmap.

The loader has not been wired up yet. It will be done in some follow-up PR. The loader has been tested within my branch.

This PR depends on NVIDIA/spark-rapids-jni#4285.

Checklists

This PR has added documentation for new or modified features or behaviors.
This PR has added new tests or modified existing tests to cover new code paths.
(Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.

greptile-apps · 2026-02-18T23:21:48Z

Greptile Summary

This PR adds a deletion vector loader for Delta Lake tables that reads serialized deletion vectors (roaring bitmaps) from disk and converts them to the standard format for GPU processing. The implementation handles two Delta serialization formats (portable and native), performing CRC validation and format conversion as needed.

Key changes:

Added RapidsDeletionVectorStore trait and RapidsHadoopDVStore implementation for loading deletion vectors from HDFS
Created DeltaSerializedBitmapLoader with format-specific loaders (DeltaPortableFormatLoader, DeltaNativeFormatLoader) that strip Delta's magic numbers and convert to standard roaring bitmap format
Added RapidsDeletionVectorStoredBitmap wrapper for deletion vector descriptors with support for empty and on-disk bitmaps
Exposed loadDeletionVector public API in RapidsDeletionVectorUtils with filter type validation

Notes:

Code properly uses closeOnExcept and withResource patterns for resource management throughout
Native format conversion requires deserialization and re-serialization (acknowledged as sub-optimal but acceptable for rare legacy format)
Previous review concerns about resource leaks, pattern matching, and CRC validation have been addressed
Depends on spark-rapids-jni PR Add support for regexp_extract on the GPU #4285 for Hash.hostCrc32 method
Not yet wired into the execution path (planned for follow-up PR)

Confidence Score: 4/5

Safe to merge with low risk - well-structured code with proper error handling, though not yet active in execution path
Code demonstrates good engineering practices with proper resource management, CRC validation, and meaningful error messages. Previous review issues have been addressed. Score is 4 rather than 5 because: (1) the loader is not yet wired up so runtime behavior is untested in this PR, (2) depends on external JNI PR, and (3) no tests included in this PR (though tested in separate branch)
No files require special attention - all three files show consistent quality and proper patterns

Important Files Changed

Filename	Overview
delta-lake/common/src/main/delta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala	New deletion vector loader with format conversion (portable/native to standard), includes CRC validation and proper resource management
delta-lake/common/src/main/delta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsStoredBitmap.scala	Simple wrapper for deletion vector descriptors with proper resource handling for empty and on-disk bitmaps
delta-lake/common/src/main/delta-33x-40x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase.scala	Added public API to load deletion vectors from descriptors with filter type validation

_{Last reviewed commit: fdcb9d2}

greptile-apps

_{3 files reviewed, 7 comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

greptile-apps

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

…k/sql/delta/deletionvectors/RapidsStoredBitmap.scala Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

greptile-apps

_{3 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

liurenjie1024 · 2026-02-25T08:28:47Z

+    dvDescriptor: DeletionVectorDescriptor,
+    tableDataPath: Path
+) {
+  require(dvDescriptor.isOnDisk, "Only on-disk deletion vectors are supported")


Is this check necessary? I think embeded deletetion vector should be processed similarly.

My understanding is that the inline deletion vectors are used only for the CRC reader or when the bitmap is empty, but are never used to store non-empty bitmaps. I don't think we should support something that is never used. The empty bitmap is being handled in the load() function of this class. As for the CRC reader, see where DeletionVectorDescriptor.inlineInLog is called.

I thought it's used for small bitmap that doesn't need to land in disk. Anyway, we could do it in a follow up if necessary.

I double-checked the Delta code base as @gerashegalov mentioned this as well, but I don't see any code using the inline deletion vector besides the CRC reader and the empty bitmap. I also did a simple experiment that I made a small deletion vector with one row deleted for the TPC-DS web_sales table at sf=100, but Delta did make a deletion vector file. So, I'm quite convinced now that the inline deletion vector is never used for our use case. That being said, I could be still wrong and we will fix this if this turns out to be an issue.

…-loader

greptile-apps

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

jihoonson · 2026-02-26T18:18:36Z

build

…-loader

jihoonson · 2026-02-26T21:57:50Z

build

jihoonson · 2026-02-26T22:43:24Z

build

jihoonson · 2026-02-26T23:32:22Z

build

gerashegalov

LGTM, nits

jihoonson · 2026-02-27T06:22:11Z

build

jihoonson · 2026-02-27T18:16:48Z

@gerashegalov @liurenjie1024 thanks for the review!

Fixes NVIDIA#14263 ### Description This PR adds a deletion vector loader for Delta scan that loads serialized deletion vectors (roaring bitmaps) into host memory. As Delta serialization formats are different from the standard serialization format described in NVIDIA#14263, the loader performs the format conversion and returns a `HostMemoryBuffer` containing the bitmap serialized in the standard format. Note that many pieces of the code are adopted from Delta classes such as `DeletionVectorStore` and `StoredBitmap`. The loader has not been wired up yet. It will be done in some follow-up PR. The loader has been tested within my [branch](https://github.com/jihoonson/spark-rapids/tree/native-dv). This PR depends on NVIDIA/spark-rapids-jni#4285. ### Checklists - [ ] This PR has added documentation for new or modified features or behaviors. - [ ] This PR has added new tests or modified existing tests to cover new code paths. (Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.) - [ ] Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description. --------- Signed-off-by: Jihoon Son <ghoonson@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

jihoonson force-pushed the dv-loader branch from e92db91 to 6b2c22a Compare February 18, 2026 23:19

jihoonson marked this pull request as draft February 18, 2026 23:20

greptile-apps Bot reviewed Feb 18, 2026

View reviewed changes

jihoonson added 3 commits February 20, 2026 11:07

Add deletion vector loader

eddf49f

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

fix buffer leak on except

39225f1

remove redundant isInline check

efe14da

jihoonson force-pushed the dv-loader branch from bee303b to efe14da Compare February 20, 2026 19:08

jihoonson marked this pull request as ready for review February 20, 2026 19:08

greptile-apps Bot reviewed Feb 20, 2026

View reviewed changes

fix build failure

849bd5c

greptile-apps Bot reviewed Feb 20, 2026

View reviewed changes

Comment thread ...main/delta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsStoredBitmap.scala Outdated

Update delta-lake/common/src/main/delta-33x-40x/scala/org/apache/spar…

87d9c44

…k/sql/delta/deletionvectors/RapidsStoredBitmap.scala Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

greptile-apps Bot reviewed Feb 20, 2026

View reviewed changes

Comment thread ...main/delta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsStoredBitmap.scala

Comment thread ...main/delta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsStoredBitmap.scala Outdated

fix build

7dc2c32

greptile-apps Bot reviewed Feb 21, 2026

View reviewed changes

jihoonson self-assigned this Feb 24, 2026

sameerz requested a review from liurenjie1024 February 24, 2026 22:35

jihoonson requested a review from a team February 24, 2026 22:36

gerashegalov reviewed Feb 25, 2026

View reviewed changes

liurenjie1024 reviewed Feb 25, 2026

View reviewed changes

jihoonson added 3 commits February 25, 2026 10:22

address comments

d5b839c

use RapidsFileIO

9c6392d

Merge branch 'dv-loader' of github.com:jihoonson/spark-rapids into dv…

1eb6bd9

…-loader

greptile-apps Bot reviewed Feb 25, 2026

View reviewed changes

Comment thread ...lta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala

import order

34073f9

greptile-apps Bot reviewed Feb 25, 2026

View reviewed changes

sameerz added the task Work required that improves the product but is not user facing label Feb 25, 2026

Merge branch 'main' of github.com:NVIDIA/spark-rapids into dv-loader

eff435b

Merge branch 'dv-loader' of github.com:jihoonson/spark-rapids into dv…

e32d7a5

…-loader

greptile-apps Bot reviewed Feb 26, 2026

View reviewed changes

Comment thread ...lta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala

Comment thread ...lta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala

add the missing default handler for fileIO pattern matching

36e7ece

greptile-apps Bot reviewed Feb 26, 2026

View reviewed changes

Comment thread ...lta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala

Comment thread ...lta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala

fix double-read of magic number for native format

cdbf29f

gerashegalov previously approved these changes Feb 27, 2026

View reviewed changes

Comment thread ...delta-33x-40x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase.scala Outdated

Comment thread ...lta-33x-40x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala

improve error message for invalid filter types

fdcb9d2

jihoonson dismissed gerashegalov’s stale review via fdcb9d2 February 27, 2026 06:18

liurenjie1024 approved these changes Feb 27, 2026

View reviewed changes

jihoonson merged commit 8b6e24c into NVIDIA:main Feb 27, 2026
45 checks passed

Conversation

jihoonson commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

greptile-apps Bot commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

greptile-apps Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liurenjie1024 Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

jihoonson Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

jihoonson Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Feb 26, 2026

Uh oh!

jihoonson commented Feb 26, 2026

Uh oh!

Uh oh!

Uh oh!

jihoonson commented Feb 26, 2026

Uh oh!

Uh oh!

Uh oh!

jihoonson commented Feb 26, 2026

Uh oh!

gerashegalov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jihoonson commented Feb 18, 2026 •

edited

Loading

greptile-apps Bot commented Feb 18, 2026 •

edited

Loading

greptile-apps Bot left a comment •

edited

Loading