Skip to content

[SPARK-58948][SQL] Deduplicate nulls-buffer creation in columnar decompressors - #58228

Open
uros-b wants to merge 3 commits into
apache:masterfrom
uros-b:helper-compressionschemes-nullindex
Open

[SPARK-58948][SQL] Deduplicate nulls-buffer creation in columnar decompressors#58228
uros-b wants to merge 3 commits into
apache:masterfrom
uros-b:helper-compressionschemes-nullindex

Conversation

@uros-b

@uros-b uros-b commented Aug 23, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Each columnar decompressor repeats the same two-line idiom to build the "nulls buffer" it reads null-tracking state from:

val nullsBuffer = buffer.duplicate().order(ByteOrder.nativeOrder())
nullsBuffer.rewind()

This adds a createNullsBuffer(buffer: ByteBuffer) helper to the CompressionScheme companion object (alongside the existing columnHeaderSize helper) and routes the six call sites through it (PassThrough, RunLengthEncoding, DictionaryEncoding, BooleanBitSet, IntDelta, LongDelta). The subsequent nullCount / nextNullIndex reads stay inline, since the nextNullIndex sentinel legitimately differs (-1, and capacity for PassThrough).

Why are the changes needed?

The idiom was duplicated across all six decoders. Naming it once removes the duplication and gives a single home for the "duplicate, set native order, rewind" convention. It also normalizes the LongDelta site, which called rewind without parentheses.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing columnar compression tests. createNullsBuffer returns exactly the previously-inlined expression, so decompression behavior is unchanged.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

@uros-b
uros-b requested a review from dongjoon-hyun August 23, 2026 21:39

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for review @dongjoon-hyun

@dongjoon-hyun dongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants