Skip to content

Support BYTES array literals end to end - #19247

Draft
xiangfu0 wants to merge 1 commit into
apache:masterfrom
xiangfu0:xiangfu0/bytes-array-support
Draft

Support BYTES array literals end to end#19247
xiangfu0 wants to merge 1 commit into
apache:masterfrom
xiangfu0:xiangfu0/bytes-array-support

Conversation

@xiangfu0

Copy link
Copy Markdown
Contributor

Summary

  • Parse SQL binary literals such as X'0102' as BYTES and construct BYTES_ARRAY values in both query engines.
  • Support bare literal projection, literal composition, and predicates that compare an ingested multi-value BYTES column with a BYTES array literal.
  • Normalize the multi-stage runtime representation from byte[][] to ByteArray[] before block serialization.
  • Add focused parser, transform, broker, planner-serde, and runtime tests plus an ingestion/query E2E test covering dictionary and raw forward indexes.

Usage

Declare a stored multi-value BYTES column with the existing Pinot schema representation:

{
  "name": "byte_values",
  "dataType": "BYTES",
  "singleValueField": false
}

Construct a BYTES array with standard SQL hex binary literals:

SELECT ARRAY[X'00', X'0102', X'FF'] AS byte_values;

Use the literal with an ingested multi-value BYTES column:

SELECT id, byte_values
FROM events
WHERE ARRAYS_OVERLAP(byte_values, ARRAY[X'0102', X'CAFE']);

Use X'CAFE' for bytes rather than the string 'CAFE'. Query response values are hex strings, and result metadata reports BYTES_ARRAY.

Compatibility

This change adds no Thrift or protobuf fields. The single-stage engine retains the existing arrayValueConstructor expression with scalar binaryValue operands on the broker/server wire, while the multi-stage engine uses its existing BYTES_ARRAY protobuf representation. This avoids introducing a new literal union arm that older servers cannot decode.

Validation

  • BytesMvTypeTest: 14/14 passed, covering Avro array<bytes> ingestion, dictionary and raw storage, projection, literal-only queries, and ARRAYS_OVERLAP under both query engines.
  • LiteralOnlyBrokerRequestTest, RequestUtilsTest, and ArrayLiteralTransformFunctionTest: passed in a clean focused reactor build.
  • RexExpressionSerDeTest: 20/20 passed.
  • TransformOperatorTest: 8/8 passed.
  • spotless:apply, checkstyle:check, license:format, and license:check: passed for all six affected modules.

The full test-compile -Xlint:all reactor check stops in unchanged pinot-segment-local because ZstandardDecompressor cannot resolve org.jetbrains.annotations.NotNull under JDK 25. All affected sources were compiled by the focused green test runs above, and no warning was emitted for an added line before that baseline failure.

@codecov-commenter

codecov-commenter commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 44.59459% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.04%. Comparing base (cb89c0e) to head (e3c4ad1).

Files with missing lines Patch % Lines
...nsform/function/ArrayLiteralTransformFunction.java 0.00% 29 Missing ⚠️
.../pinot/query/runtime/operator/utils/TypeUtils.java 0.00% 6 Missing ⚠️
...sthandler/BaseSingleStageBrokerRequestHandler.java 75.00% 1 Missing and 2 partials ⚠️
.../parsers/rewriter/CompileTimeFunctionsInvoker.java 88.88% 1 Missing and 1 partial ⚠️
...e/pinot/common/function/scalar/ArrayFunctions.java 80.00% 0 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (cb89c0e) and HEAD (e3c4ad1). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (cb89c0e) HEAD (e3c4ad1)
unittests1 1 0
unittests 2 1
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #19247       +/-   ##
=============================================
- Coverage     66.97%   39.04%   -27.94%     
- Complexity     1417     1422        +5     
=============================================
  Files          3453     3453               
  Lines        218858   218927       +69     
  Branches      34787    34802       +15     
=============================================
- Hits         146591    85488    -61103     
- Misses        60556   125619    +65063     
+ Partials      11711     7820     -3891     
Flag Coverage Δ
integration 100.00% <ø> (+100.00%) ⬆️
integration1 100.00% <ø> (?)
integration2 0.00% <ø> (ø)
java-25 39.04% <44.59%> (-27.94%) ⬇️
lane-a 100.00% <ø> (+100.00%) ⬆️
lane-b 0.00% <ø> (ø)
temurin 39.04% <44.59%> (-27.94%) ⬇️
unittests 39.04% <44.59%> (-27.94%) ⬇️
unittests1 ?
unittests2 39.04% <44.59%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants