Skip to content

[WIP] Bloom filter implementation - #9398

Draft
joacoc wants to merge 5 commits into
vortex-data:developfrom
joacoc:bloom-index
Draft

[WIP] Bloom filter implementation#9398
joacoc wants to merge 5 commits into
vortex-data:developfrom
joacoc:bloom-index

Conversation

@joacoc

@joacoc joacoc commented Aug 13, 2026

Copy link
Copy Markdown

Rationale for this change

This is part of a bigger issue around skip indexes but only focuses on the Bloom filter implementation:

What changes are included in this PR?

This is a continuation of a previous PR that already implements a simpler version of a bloom filter and also includes the writer interface for skip indexes: #8933.

Most of the changes included in this PR are replacing the hand-rolled bloom-filter from the base PR with the split block bloom filter, and expanding support for more canonicals and constants. So far it supports bool, primitives, decimals, varbinview, and extensions, and this PR leaves out composable types out of the implementation (struct, list and so on).

What APIs are changed? Are there any user-facing changes?

The writer user interface includes the skip index builder from the base PR #8933. This is included in the draft as a way to integrate and test the whole flow but I haven't put much thought yet into how it should behave or look like.

About the code

The Bloom filter implementation is a translation from the paper source code, given that it is very short I thought it wouldn't be a big deal. The code doesn't use arch-y fn calls, but relies on the compiler optimizer for vectorizing/calling the correct assembler expression, which from my checks on godbolt, seems to get translated well. I could have gone into specializing the code into different architectures but keeping it as is seemed better for maintenance.

Accumulators

Bloom filter's accumulators for canonicals are very similar to those of the min/max aggregation fn. I used those implementations as a reference for manipulating each case, with some slight variations.

Invalid values are left out of Bloom filters. Only valid values apply. Accumulating an invalid value will just skip it, but asking for a scalar that is invalid will raise a VortexError.

Options/tuning

The Bloom filter has a single tuning option called blocks_count. The more blocks, the more memory/space usage (per zone), but the fewer false positives. Defaults to 256 (8KiB p/zone), derived from the default amount of 8192 rows in a zone. The paper suggests keeping between 20 and 52 distinct hash values per block. Assuming all 8,192 values are distinct, choosing 32 values per block gives 8,192 / 32 = 256 blocks (low false positive rate: ~3%).

Another approach could be to use statistics from a subset of the data, as is done with encoders, to select the best block count. I think this approach aligns with how Vortex encoders work, but it shouldn't be part of this PR, as it may depend on how the skip index writer is designed.

Hasher crate

This PR also includes a new crate twox-hash that implements XxHash, similar to how Parquet uses the same hasher for the same use case, but it relies on vendored code. Happy to change or look for alternatives.

connortsui20 and others added 5 commits August 14, 2026 12:17
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…ion comments and fix primitive test for validation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants