Skip to content

feat: support ALP parquet encoding in writer options #27036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 2 commits into from
Nov 3, 2026
+48 −9
Merged

feat: support ALP parquet encoding in writer options #27036

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1f1278f
feat: support ALP parquet encoding in writer options
Surajsuthar Nov 3, 2026
86b0f5a
Merge branch 'main' into feat/added-alp-write-support
Surajsuthar Nov 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions datafusion/common/src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1512,7 +1512,7 @@ config_namespace! {
/// (writing) Sets default encoding for any column.
/// Valid values are: plain, plain_dictionary, rle,
/// bit_packed, delta_binary_packed, delta_length_byte_array,
/// delta_byte_array, rle_dictionary, and byte_stream_split.
/// delta_byte_array, rle_dictionary, byte_stream_split, and alp.
/// These values are not case sensitive. If NULL, uses
/// default parquet writer setting
pub encoding: Option<DFParquetEncoding>, default = None
Expand Down Expand Up @@ -3509,7 +3509,7 @@ config_namespace_with_hashmap! {
/// Sets encoding for the column path.
/// Valid values are: plain, plain_dictionary, rle,
/// bit_packed, delta_binary_packed, delta_length_byte_array,
/// delta_byte_array, rle_dictionary, and byte_stream_split.
/// delta_byte_array, rle_dictionary, byte_stream_split, and alp.
/// These values are not case-sensitive. If NULL, uses
/// default parquet options
pub encoding: Option<String>, default = None
Expand Down
8 changes: 7 additions & 1 deletion datafusion/common/src/parquet_config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,8 @@ pub enum DFParquetEncoding {
RleDictionary,
/// Byte stream split encoding
ByteStreamSplit,
/// Adaptive Lossless Floating-Point Encoding
Alp,
}

impl FromStr for DFParquetEncoding {
Expand All @@ -301,11 +303,12 @@ impl FromStr for DFParquetEncoding {
"delta_byte_array" => Ok(Self::DeltaByteArray),
"rle_dictionary" => Ok(Self::RleDictionary),
"byte_stream_split" => Ok(Self::ByteStreamSplit),
"alp" => Ok(Self::Alp),
_ => Err(DataFusionError::Configuration(format!(
"Unknown or unsupported parquet encoding: \
{s}. Valid values are: plain, plain_dictionary, rle, \
bit_packed, delta_binary_packed, delta_length_byte_array, \
delta_byte_array, rle_dictionary, and byte_stream_split."
delta_byte_array, rle_dictionary, byte_stream_split, and alp."
))),
}
}
Expand All @@ -323,6 +326,7 @@ impl Display for DFParquetEncoding {
Self::DeltaByteArray => "delta_byte_array",
Self::RleDictionary => "rle_dictionary",
Self::ByteStreamSplit => "byte_stream_split",
Self::Alp => "alp",
})
}
}
Expand Down Expand Up @@ -374,6 +378,7 @@ impl From<DFParquetEncoding> for parquet::basic::Encoding {
DFParquetEncoding::DeltaByteArray => Self::DELTA_BYTE_ARRAY,
DFParquetEncoding::RleDictionary => Self::RLE_DICTIONARY,
DFParquetEncoding::ByteStreamSplit => Self::BYTE_STREAM_SPLIT,
DFParquetEncoding::Alp => Self::ALP,
}
}
}
Expand Down Expand Up @@ -737,6 +742,7 @@ mod tests {
DFParquetEncoding::ByteStreamSplit,
Encoding::BYTE_STREAM_SPLIT,
),
("alp", DFParquetEncoding::Alp, Encoding::ALP),
] {
let parsed: DFParquetEncoding = input.parse().unwrap();
assert_eq!(parsed, expected, "input: {input}");
Expand Down
6 changes: 5 additions & 1 deletion datafusion/proto-common/src/from_proto/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1723,7 +1723,11 @@ mod tests {

#[test]
fn test_parquet_encoding_round_trip() {
for encoding in [None, Some(DFParquetEncoding::ByteStreamSplit)] {
for encoding in [
None,
Some(DFParquetEncoding::ByteStreamSplit),
Some(DFParquetEncoding::Alp),
] {
let opts = ParquetOptions {
encoding,
..ParquetOptions::default()
Expand Down
21 changes: 21 additions & 0 deletions datafusion/sqllogictest/test_files/copy.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,27 @@ OPTIONS (
----
2

# ALP encoding is accepted for floating-point columns.
query I
COPY (
VALUES (CAST(1.25 AS DOUBLE)), (CAST(2.5 AS DOUBLE)), (CAST(3.75 AS DOUBLE))
)
TO 'test_files/scratch/copy/alp/' STORED AS PARQUET
OPTIONS ('format.encoding' 'alp', 'format.dictionary_enabled' false);
----
3

statement ok
CREATE EXTERNAL TABLE validate_alp
STORED AS PARQUET LOCATION 'test_files/scratch/copy/alp/';

query R rowsort
SELECT column1 FROM validate_alp;
----
1.25
2.5
3.75

# valid vs invalid metadata

# accepts map with a single entry
Expand Down
2 changes: 1 addition & 1 deletion datafusion/sqllogictest/test_files/information_schema.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -410,7 +410,7 @@ datafusion.execution.parquet.dictionary_enabled true (writing) Sets if dictionar
datafusion.execution.parquet.dictionary_page_size_limit 1048576 (writing) Sets best effort maximum dictionary page size, in bytes
datafusion.execution.parquet.enable_page_index true (reading) If true, reads the Parquet data page level metadata (the Page Index), if present, to reduce the I/O and number of rows decoded.
datafusion.execution.parquet.enable_rle_to_dictionary false (reading) If true, top-level string and binary Parquet columns with dictionary pages are inferred and scanned as `Dictionary<Int32, Utf8>` / `Dictionary<Int32, Binary>` instead of their plain value type. This applies only when DataFusion infers the table schema. Tables with a user-supplied schema are not promoted because the Parquet footer is not read at DDL time, so dictionary pages cannot be detected per column. See <https://github.com/apache/datafusion/issues/24112>
datafusion.execution.parquet.encoding NULL (writing) Sets default encoding for any column. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, and byte_stream_split. These values are not case sensitive. If NULL, uses default parquet writer setting
datafusion.execution.parquet.encoding NULL (writing) Sets default encoding for any column. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, byte_stream_split, and alp. These values are not case sensitive. If NULL, uses default parquet writer setting
datafusion.execution.parquet.force_filter_selections false (reading) Force the use of RowSelections for filter results, when pushdown_filters is enabled. If false, the reader will automatically choose between a RowSelection and a Bitmap based on the number and pattern of selected rows.
datafusion.execution.parquet.max_in_list_size 20 Maximum number of input values in an `IN (...)` list eligible for min/max pruning. Lists above this cap, or a cap of 0, skip this rewrite; other predicates and Bloom-filter pruning remain available. Within the cap, nonempty lists of at most 20 values use the existing per-value rewrite. Larger literal lists use a compact representation when the column type is string, variable-length binary, integer, decimal, date, time, timestamp, or duration. This applies to both `IN` and `NOT IN`, including lists with NULL members. `NOT IN` with NULL and all-NULL `IN` lists cannot match any rows. Compact lists containing NULL do not use the fully-matched-row-group optimization. Floating-point and other lists retain the existing per-value rewrite, so raising the cap can make those predicates expensive to build and evaluate. Defaults to 20.
datafusion.execution.parquet.max_predicate_cache_size NULL (reading) The maximum predicate cache size, in bytes. When `pushdown_filters` is enabled, sets the maximum memory used to cache the results of predicate evaluation between filter evaluation and output generation. Decreasing this value will reduce memory usage, but may increase IO and CPU usage. None means use the default parquet reader setting. 0 means no caching.
Expand Down
12 changes: 10 additions & 2 deletions datafusion/sqllogictest/test_files/set_variable.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -754,18 +754,26 @@ SHOW datafusion.execution.parquet.encoding
----
datafusion.execution.parquet.encoding plain

statement ok
SET datafusion.execution.parquet.encoding = 'ALP'

query TT
SHOW datafusion.execution.parquet.encoding
----
datafusion.execution.parquet.encoding alp

statement error
SET datafusion.execution.parquet.encoding = 'invalid'
----
DataFusion error: Error setting config datafusion.execution.parquet.encoding
caused by
Invalid or Unsupported Configuration: Unknown or unsupported parquet encoding: invalid. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, and byte_stream_split.
Invalid or Unsupported Configuration: Unknown or unsupported parquet encoding: invalid. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, byte_stream_split, and alp.


query TT
SHOW datafusion.execution.parquet.encoding
----
datafusion.execution.parquet.encoding plain
datafusion.execution.parquet.encoding alp

statement ok
RESET datafusion.execution.parquet.encoding
Expand Down
2 changes: 1 addition & 1 deletion docs/source/user-guide/configs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ The following configuration settings are available:
| datafusion.execution.parquet.column_index_truncate_length | 64 | (writing) Sets column index truncate length |
| datafusion.execution.parquet.statistics_truncate_length | 64 | (writing) Sets statistics truncate length. If NULL, uses default parquet writer setting |
| datafusion.execution.parquet.data_page_row_count_limit | 20000 | (writing) Sets best effort maximum number of rows in data page |
| datafusion.execution.parquet.encoding | NULL | (writing) Sets default encoding for any column. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, and byte_stream_split. These values are not case sensitive. If NULL, uses default parquet writer setting |
| datafusion.execution.parquet.encoding | NULL | (writing) Sets default encoding for any column. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, byte_stream_split, and alp. These values are not case sensitive. If NULL, uses default parquet writer setting |
| datafusion.execution.parquet.bloom_filter_on_write | false | (writing) Write bloom filters for all columns when creating parquet files |
| datafusion.execution.parquet.bloom_filter_fpp | NULL | (writing) Sets bloom filter false positive probability. If NULL, uses default parquet writer setting |
| datafusion.execution.parquet.bloom_filter_ndv | NULL | (writing) Sets bloom filter number of distinct values. If NULL, uses default parquet writer setting |
Expand Down
Loading
Loading