[Bug] be/test does not compile on macOS: Linux-only APIs and Darwin type identity in recently added tests

[robll-v1]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master. Reproduced on f5da8fb165 and still present on 1a3772ff04.

Platform / toolchain: macOS 27 (build 26A428) arm64 (Apple Silicon), Homebrew LLVM/Clang 23.1.1, CMake 3.25.3, ninja 1.13.2, JDK 17, BUILD_TYPE=ASAN, third-party from the doris-thirdparty-prebuilt-darwin-arm64 release.

What's Wrong?

be/test does not compile on macOS, so run-be-ut.sh is unusable locally. Note that .github/workflows/be-ut-mac.yml deliberately does not compile be/test ("Only verify that the BE compiles on macOS; do NOT compile be/test. The unit tests are built and run locally, not in this job."), so nothing in CI catches this.

All test sources are linked into the single doris_be_test target, therefore a handful of failing files blocks every --filter= run as well.

The BE itself (./build.sh --be) builds and runs fine on macOS; only the unit-test target is affected.

7 TUs fail. 5 of them are macOS-incompatible test code added in the last few weeks:

file	compiler error	why it only fails on macOS
be/test/storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp	6 × format specifies type 'long' but the argument has type 'int64_t' (aka 'long long') / 'unsigned long' but 'uint64_t' (aka 'unsigned long long') [-Werror,-Wformat]	Darwin int64_t is long long; on glibc it is long
be/test/storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp	no member named 'posix_fadvise' in the global namespace, use of undeclared identifier 'POSIX_FADV_DONTNEED', and cannot initialize a parameter of type 'char *' with an rvalue of type 'value_type *' (aka 'unsigned char *') for mincore()	posix_fadvise() is Linux-only and mincore()'s 3rd parameter is char* on macOS
be/test/storage/index/snii/bkd/bkd_corruption_test.cpp	2 × deduced conflicting types ('size_type' (aka 'unsigned long') vs 'uint64_t' (aka 'unsigned long long')) for initializer list element type	size_t and uint64_t are distinct types on Darwin
be/test/storage/index/snii/writer/spimi_locality_bench_test.cpp	no matching function for call to object of type 'key_equal' (aka 'Equal') raised from libc++ <__hash_table>	the custom Equal functor used for heterogeneous lookup does not provide the overloads libc++ instantiates
be/test/storage/segment/hierarchical_data_iterator_test.cpp	3 × no matching function for call to 'min' (std::min(*rows, kRows - _state->current_ordinal))	uint64_t vs size_t (unsigned long long vs unsigned long)

2 more TUs fail for include/build-configuration reasons (also reproduced with the same toolchain):

file	compiler error
be/test/exec/dictionary/dictionary_ip_trie_test.cpp	be/src/core/value/ip_address_cidr.h:181: no member named 'from_chars' in namespace 'std' — the header uses std::from_chars without including <charconv>
be/test/exprs/function/function_map_inner_product_test.cpp	'faiss/impl/platform_macros.h' file not found — it includes be/src/exprs/function/array/function_array_distance.h, which needs faiss headers; be/test/CMakeLists.txt already removes storage/index/ann/*.cpp from UT_FILES for exactly this reason, but not test files that import faiss headers indirectly

Additionally, with clang ≥ 23 the __COUNTER__ uses in be/src/runtime/memory/lru_cache_policy.h and be/src/storage/segment/variant/hierarchical_data_iterator.h become hard errors under the project's -Wpedantic -Werror ('__COUNTER__' is a C2y extension [-Werror,-Wc2y-extensions]). Not macOS-specific, but it independently prevents building the UT target with a current clang.

What You Expected?

./run-be-ut.sh -j 4 and ./run-be-ut.sh --run --filter=<some_test> should build and run on macOS, as the macOS workflow's comment says unit tests are meant to be built locally.

How to Reproduce?

# macOS arm64, master
git clone https://github.com/apache/doris.git && cd doris
./build.sh --be -j 4       # succeeds, doris_be is produced
./run-be-ut.sh -j 4        # fails while compiling be/test

# a single TU is enough to see the platform issues, e.g.
cd be/ut_build_ASAN
ninja test/CMakeFiles/doris_be_test.dir/storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp.o

Exact lines / errors (paths shortened to the repository root):

===== be/test/storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp  (6 errors)
be/test/storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp:375:88: error: format specifies type 'long' but the argument has type 'int64_t' (aka 'long long') [-Werror,-Wformat]
be/test/storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp:403:12: error: format specifies type 'unsigned long' but the argument has type 'uint64_t' (aka 'unsigned long long') [-Werror,-Wformat]
be/test/storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp:404:79: error: format specifies type 'unsigned long' but the argument has type 'uint64_t' (aka 'unsigned long long') [-Werror,-Wformat]
be/test/storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp:482:44: error: format specifies type 'unsigned long' but the argument has type 'uint64_t' (aka 'unsigned long long') [-Werror,-Wformat]
be/test/storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp:582:64: error: format specifies type 'unsigned long' but the argument has type 'uint64_t' (aka 'unsigned long long') [-Werror,-Wformat]
be/test/storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp:707:79: error: format specifies type 'unsigned long' but the argument has type 'uint64_t' (aka 'unsigned long long') [-Werror,-Wformat]

===== be/test/storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp  (3 platform errors)
be/test/storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp:1048:62: error: cannot initialize a parameter of type 'char *' with an rvalue of type 'value_type *' (aka 'unsigned char *')
be/test/storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp:1083:15: error: no member named 'posix_fadvise' in the global namespace; did you mean 'posix_madvise'?
be/test/storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp:1083:39: error: use of undeclared identifier 'POSIX_FADV_DONTNEED'

===== be/test/storage/index/snii/bkd/bkd_corruption_test.cpp  (2 errors)
be/test/storage/index/snii/bkd/bkd_corruption_test.cpp:557:36: error: deduced conflicting types ('size_type' (aka 'unsigned long') vs 'uint64_t' (aka 'unsigned long long')) for initializer list element type
be/test/storage/index/snii/bkd/bkd_corruption_test.cpp:566:36: error: deduced conflicting types ('uint64_t' (aka 'unsigned long long') vs 'size_type' (aka 'unsigned long')) for initializer list element type

===== be/test/storage/index/snii/writer/spimi_locality_bench_test.cpp  (1 error)
.../c++/v1/__hash_table:1802:43: error: no matching function for call to object of type 'key_equal' (aka 'Equal')

===== be/test/storage/segment/hierarchical_data_iterator_test.cpp  (3 errors)
be/test/storage/segment/hierarchical_data_iterator_test.cpp:121:33: error: no matching function for call to 'min'
be/test/storage/segment/hierarchical_data_iterator_test.cpp:169:33: error: no matching function for call to 'min'
be/test/storage/segment/hierarchical_data_iterator_test.cpp:224:33: error: no matching function for call to 'min'

===== be/test/exec/dictionary/dictionary_ip_trie_test.cpp  (1 error)
be/src/core/value/ip_address_cidr.h:181:42: error: no member named 'from_chars' in namespace 'std'; did you mean 'to_chars'?

===== be/test/exprs/function/function_map_inner_product_test.cpp  (1 error)
be/src/exprs/function/array/function_array_distance.h:20:10: fatal error: 'faiss/impl/platform_macros.h' file not found

Anything Else?

• The failing files are recent: the snii ones were added by 96fd1e9bd2, 1db4b1d305, 9f3b084a75 (Aug–Sep 2026) and hierarchical_data_iterator_test.cpp was changed by 388c93aa8a ([feature](variant) Enable Variant V2 by default, [feature](variant) Enable Variant V2 by default #66858, 2026-09-14). macOS CI intentionally skips be/test, so these went unnoticed.
• Two possible fixes:
  1. make the 5 files portable (%lld/%llu or the PRI* macros, reinterpret_cast<char*> for mincore, #if defined(__linux__) around posix_fadvise, explicit std::min<uint64_t>/casts, add the missing Equal::operator()(uint32_t, std::string_view) or drop heterogeneous lookup for that bench, and #include <charconv> in ip_address_cidr.h); and/or
  2. if macOS coverage of those tests is not desired, extend the existing platform-exclusion block in be/test/CMakeLists.txt — if (OS_MACOSX) list(REMOVE_ITEM UT_FILES ${CMAKE_CURRENT_SOURCE_DIR}/util/system_metrics_test.cpp) endif() — with the files above.
• A local patch covering all of the above is already prepared (roughly 15 lines) and can be submitted as a PR if that direction is preferred.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Initial maintainer triage

Assessment: valid and actionable build-portability regression (high confidence). I inspected the exact reported revision 1a3772ff041630444eab49b98eaccbc90fabbe8e. I could not execute a Darwin build on this Linux host, but the reported diagnostics match the source and Darwin ABI/API differences. The impact is currently limited to building the monolithic doris_be_test target on macOS; the production BE build is not implicated.

The issue is open, unassigned, and currently has no labels.

Verified facts

• be/test/CMakeLists.txt glob-adds these sources to one doris_be_test executable, while .github/workflows/be-ut-mac.yml configures MAKE_TEST=OFF. Therefore any one compile failure blocks every filtered local UT run, and current macOS CI cannot detect it.
• On an arm64 Darwin target, Clang defines int64_t/uint64_t as long long/unsigned long long, while size_t is unsigned long. This directly explains the printf, initializer-list, and std::min failures. Use PRIi64/PRIu64 (or type-safe formatting) rather than hard-coding %lld/%llu, which would merely move the mismatch to platforms where int64_t is long.
• snii_vs_v3_benchmark_test.cpp unconditionally calls Linux-only posix_fadvise(..., POSIX_FADV_DONTNEED), and passes unsigned char* to Darwin's mincore(..., char*) signature.
• The libc++ heterogeneous-lookup error is consistent with the test's Equal: it has (string_view, uint32_t) but not (uint32_t, string_view). The production OwnedVocabEq in spimi_term_buffer.h already provides both directions and is the appropriate local pattern.
• ip_address_cidr.h uses std::from_chars but does not include <charconv>; successful Linux builds currently depend on a transitive include.
• The Faiss failure has a concrete CMake dependency-path explanation. function_map_inner_product_test.cpp includes function_inner_product.h, which includes function_array_distance.h and therefore Faiss headers. On non-Apple builds, doris_be_test links the vector_search_test target and inherits its transitive ann_index -> faiss usage requirements. The Apple branch instead links $<TARGET_FILE:vector_search_test> via -force_load; that archive path does not propagate target usage requirements to compilation of doris_be_test.
• The __COUNTER__ error is separate and not macOS-specific. The token is defined in runtime_profile.h; it is expanded by SCOPED_TIMER/SCOPED_RAW_TIMER call sites in the two reported headers. New Clang versions diagnose that extension through -Wcounter-extension under -Wpedantic, and Doris promotes it with -Werror.

Recommended direction

1. Prefer portable source fixes for the integer formatting/types, initializer lists, std::min, symmetric comparator, and direct <charconv> include. These are small correctness/build-hygiene fixes and preserve useful macOS coverage.
2. Treat snii_vs_v3_benchmark_test.cpp specially. A Darwin no-op around posix_fadvise would compile but would violate the benchmark's explicit “cold page cache” contract. Either implement a verified Darwin-equivalent cold-read path (and keep the existing mincore residency check meaningful), or exclude only this disabled benchmark TU on macOS.
3. Fix the Faiss dependency at the target/CMake level rather than excluding function_map_inner_product_test.cpp. The Apple whole-archive linkage should retain vector_search_test/ann_index/faiss usage requirements, or doris_be_test should receive the required dependency explicitly.
4. Handle __COUNTER__ as a toolchain-compatibility item, ideally with narrowly scoped handling of -Wcounter-extension for Clang rather than relaxing -Wpedantic globally.
5. Add at least a compile-only doris_be_test step to macOS CI. Running all UTs can remain a separate policy decision, but compiling the target is needed to prevent the same class of regression.

Missing information / acceptance checks

No additional information is required to accept and assign this issue. If a maintainer cannot reproduce the two configuration-sensitive failures, the only useful additions would be the corresponding ninja -v compile commands for the Faiss and __COUNTER__ TUs.

Before closing, verify a clean ./run-be-ut.sh -j 4 build on arm64 macOS with the reported Clang 23 toolchain, run representative filters for the enabled affected tests, and confirm the Linux BE UT build remains clean. The author's prepared patch is a reasonable starting point, but the cold-cache semantics and Apple CMake usage-requirement propagation should be reviewed as above.

Breakwater-GitHub-Analysis-Slot: slot_088b0c303f7b

[robll-v1]

Thanks for the triage — confirming the two configuration-sensitive failures with concrete evidence, plus one correction.

1) Faiss include on Apple: confirmed, and it is the -force_load code path

be/test/CMakeLists.txt:168-180 is exactly as described: the Apple branch passes the archive as a raw link flag

if (APPLE)
    target_link_libraries(doris_be_test ${TEST_LINK_LIBS}
        -Wl,-force_load,$<TARGET_FILE:vector_search_test>)
    add_dependencies(doris_be_test vector_search_test)
else()
    target_link_libraries(doris_be_test ${TEST_LINK_LIBS}
        -Wl,--whole-archive vector_search_test -Wl,--no-whole-archive)
endif()

so vector_search_test's INTERFACE usage requirements (ann_index -> faiss include dirs) never reach doris_be_test compilation on Apple, while the non-Apple branch passes vector_search_test as a target name and does propagate them.

The failing TU's recorded command (from be/ut_build_ASAN/compile_commands.json created by run-be-ut.sh) abridges to:

... -I<repo>/be/src -I<repo>/be/test ... -isystem <repo>/thirdparty/installed/include ...

with no faiss include directory anywhere, matching that explanation.

2) __COUNTER__: tag name and blast radius

With clang 23.1.1 the diagnostic is -Wc2y-extensions on this revision, not -Wcounter-extension:

$ clang++ -std=gnu++20 -Wpedantic -Werror -c counter.cpp
counter.cpp:1:18: error: '__COUNTER__' is a C2y extension [-Werror,-Wc2y-extensions]

$ clang++ -std=gnu++20 -Wpedantic -Werror -Wno-counter-extension -c counter.cpp
error: unknown warning option '-Wno-counter-extension'; did you mean '-Wno-openacc-extension'? [-Werror,-Wunknown-warning-option]

Also, this is not limited to be/test: with clang 23 the production BE build fails the same way — ./build.sh --be stops at be/src/runtime/memory/lru_cache_policy.h:177 and be/src/storage/segment/variant/hierarchical_data_iterator.h:174 (SCOPED_TIMER/SCOPED_RAW_TIMER expand __COUNTER__), and a syntax-only sweep of be/src reports ~642 sites in total.

One implementation note for whoever fixes it: adding -Wno-* to CMAKE_CXX_FLAGS/EXTRA_CXX_FLAGS is ineffective, because be/CMakeLists.txt appends -Wpedantic later on the command line and that re-enables the diagnostic group (last match wins). A #pragma clang diagnostic ignored or a flag appended after the project's -Wpedantic is required.

3) snii_vs_v3_benchmark_test.cpp

Agreed — a Darwin no-op posix_fadvise would silently invalidate the benchmark's cold-page-cache contract, so it is not an acceptable upstream fix. I only used that shim locally as a temporary probe to enumerate the remaining failing TUs. Excluding this disabled benchmark on macOS (or implementing a verified Darwin cold-read path) is the right call.

4) macOS CI

Also agree that at least a compile-only doris_be_test step in the macOS workflow would be worth it — that is the only thing that would have caught this class of regression, given the job deliberately sets -DMAKE_TEST=OFF since #65796.

Happy to prepare a PR along the lines of your option 1 (PRIu64-style formatting, symmetric comparator mirroring OwnedVocabEq in spimi_term_buffer.h, explicit std::min<uint64_t>, #include <charconv> in ip_address_cidr.h), and to either fix the Apple -force_load linkage or exclude snii_vs_v3_benchmark_test.cpp on macOS as you prefer.