[Bug] BE UT crashes under ARM64 + LSAN build due to allocation size exceeding 4GB limit

[heguanhui]

Version

trunk (master branch)

What's Wrong?

When running BE UT with CMAKE_BUILD_TYPE=LSAN on ARM64 architecture, the test cases IntersectOperatorTest::test_sink_large_string_data_over_4g and ExceptOperatorTest::test_sink_large_string_data_over_4g crash the entire UT process with the following error:

==15521==ERROR: LeakSanitizer: requested allocation size 0x200000000 exceeds maximum supported size of 0x100000000
    #0 ... in realloc
    #1 ... in doris::DefaultMemoryAllocator::realloc(void*, unsigned long) allocator.h:100
    #2 ... in doris::Allocator::realloc(...) allocator.cpp:411
    #3 ... in doris::PODArrayBase::realloc(...) pod_array.h:191
    #4 ... in doris::PODArrayBase::reserve(...) pod_array.h:261
    #5 ... in doris::PODArrayBase::resize(...) pod_array.h:267
    #6 ... in doris::ColumnStr::insert_range_from_ignore_overflow(...) column_string.cpp:113
    #7 ... in doris::MutableBlock::merge_impl_ignore_overflow(...) block.h:586
    #8 ... in doris::MutableBlock::merge_ignore_overflow(...) block.h:564
    #9 ... in doris::SetSinkOperatorX<true>::sink(...) set_sink_operator.cpp:89
    #10 ... in IntersectOperatorTest_test_sink_large_string_data_over_4g_Test::TestBody() set_operator_test.cpp:480

After this error, the UT process exits immediately, preventing any subsequent tests from running.

What You Expected?

The UT process should not crash. Tests that are incompatible with LSAN's ARM64 allocation limit should be gracefully skipped so that the rest of the UT suite can continue running.

How to Reproduce?

1. Build BE UT on ARM64 with CMAKE_BUILD_TYPE=LSAN
2. Run: ./run-be-ut.sh --run --filter=IntersectOperatorTest.test_sink_large_string_data_over_4g
3. Observe the crash

Root Cause Analysis

The crash is caused by the interaction of two factors:

1. LSAN's ARM64 allocation limit

In LSAN's source code (compiler-rt/lib/lsan/lsan_allocator.cpp), the maximum allowed single allocation size is defined per architecture:

#if defined(__i386__) || defined(__arm__)
static const uptr kMaxAllowedMallocSize = 1ULL << 30;       // 1GB
#elif defined(__mips64) || defined(__aarch64__)
static const uptr kMaxAllowedMallocSize = 4ULL << 30;       // 4GB (ARM64)
#else
static const uptr kMaxAllowedMallocSize = 1ULL << 40;       // 1TB (x86_64)
#endif

On ARM64 (__aarch64__), kMaxAllowedMallocSize = 0x100000000 (4GB). Any single allocation request exceeding this limit triggers a fatal error.

2. PODArray's power-of-two rounding

PODArrayBase::reserve() (pod_array.h:259-262) rounds up the requested size to the next power of two via round_up_to_power_of_two_or_zero(). When the test accumulates ~4.1GB of string data in ColumnStr::chars, the resize request of ~4.1GB gets rounded up to 8GB (0x200000000), which exceeds LSAN's 4GB limit on ARM64.

The detailed calculation for the Intersect test (4200 rows × 1MB per row, batched at 500 rows):

Batch	chars.size()	resize target	round_up_to_2^N	Realloc?
5	1.95GB	2.44GB	0x100000000 (4GB)	Yes
6-8	1.95~3.91GB	2.44~3.91GB	0x100000000 (4GB)	No (capacity sufficient)
9	3.91GB	4.10GB	0x200000000 (8GB)	Yes → CRASH

This issue does not occur on x86_64 because LSAN's limit there is 1TB.

Are you willing to submit PR?

• Yes

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_0e8b225c034a

Thanks for the detailed report. I checked the current master code path, and the diagnosis looks consistent.

Assessment:

• This is best treated as an ARM64 + LSAN test-environment incompatibility, not as evidence that the set-operator overflow conversion logic is broken.
• In be/test/exec/operator/set_operator_test.cpp, both affected tests build 4,200 rows of 1 MiB strings. The build side is accumulated before the final overflow conversion step.
• SetSinkOperatorX::sink() first merges incoming build blocks through MutableBlock::merge_ignore_overflow(). That reaches ColumnStr::insert_range_from_ignore_overflow(), which resizes the destination chars buffer.
• PODArrayBase::reserve() rounds the requested allocation up to the next power of two before realloc. Once the logical chars size crosses slightly over 4 GiB, the allocation request can become 8 GiB.
• LLVM LeakSanitizer limits a single allocation to 4 GiB on __aarch64__, so the process aborts before the test reaches _process_build_block() / convert_column_if_overflow().

Recommended next step:

• Add a targeted GTEST_SKIP() at the start of both test_sink_large_string_data_over_4g tests when defined(__aarch64__) && defined(LEAK_SANITIZER).
• Keep the tests enabled for x86_64 LSAN and for non-LSAN ARM64 builds, because they still cover the intended over-4GiB ColumnString conversion path where the allocator permits it.
• I would not change PODArray growth behavior for this issue. Its power-of-two reserve policy is shared by many vector columns, and changing it would have broad performance and memory-behavior impact for a sanitizer-specific synthetic UT limit.

Useful verification details to include in the PR:

• The exact compiler / LLVM version used by the failing ARM64 LSAN build.
• The full BE UT command and the first sanitizer error line showing the 8 GiB request versus the 4 GiB limit.
• Confirmation that the tests still run on x86_64 LSAN and on ARM64 non-LSAN builds after adding the targeted skip.