Skip to content

Add bundled native release packages and installed artifact tests - #1016

Open
ColinLeeo wants to merge 6 commits into
apache:developfrom
ColinLeeo:colin/native-release-artifacts-develop
Open

ColinLeeo wants to merge 6 commits into
apache:developfrom
ColinLeeo:colin/native-release-artifacts-develop

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Native C++ releases need installable SDKs and platform packages with consistent bundled dependencies, and downloaded artifacts need validation before publication. This adds one manually dispatched workflow that builds native packages and Python wheels, tests their installed contents, and assembles a checksummed bundle only after every required platform succeeds.

Changes

  • Export relocatable CMake and pkg-config SDK metadata, the compatibility header closure, static dependencies, and resolved Windows CRT/runtime information for independent C++, Go and Python consumers. Static pkg-config consumers receive private codec/platform/C++ runtime dependencies with configuration-specific archive paths.
  • Build native releases with pinned, checksum-verified BUNDLED dependencies. Make bundled dependencies the CMake/Maven default, remove Homebrew codec dependencies, and reject external third-party dylib references in macOS bottles and SDKs.
  • Produce 12 SDK variants, Ubuntu amd64/arm64 DEBs, AlmaLinux 9 x86_64/aarch64 RPMs, ARM64/Intel Homebrew bottles with a merged tsfile Formula, Windows x86_64/ARM64 ZIPs, and 30 Python wheels covering CPython 3.9–3.14 on five platforms. Reuse staged SDKs instead of rebuilding the C++ core for every wheel.
  • Keep the CLI build-tree RPATH separate from its relative installed RPATH; validate both lib64 and multiarch layouts after relocating the SDK and deleting the build directory. Keep exported path-parser compile definitions consistent with the library objects to avoid Linux Debug UTF-8 failures.
  • Download and install the outputs in separate jobs. Run existing C++ API regression tests through the installed CMake package, CLI operations on four existing TsFile fixtures, Go vet/tests and supported race checks, and the full pytest suite against every installed wheel. Relocate and test every downloadable SDK archive.
  • Generate manifest.json and SHA256SUMS with exact source identity, versions, sizes and hashes. C++ publication targets use the release paths/components in tsfile, tsfile-debian and tsfile-rpm; Python entries contain no Apache repository target and retain the separate PyPI publication process.
  • Ship runtime-only wheels with one TsFile library, stripped Unix binaries and license texts, while preserving the build inputs needed in source distributions. Replace the separate wheel-build workflow with the unified native workflow.

The workflow is release-only: it accepts MAJOR.MINOR.PATCH source versions and rejects development/prerelease versions before building. It runs only through workflow_dispatch, has read-only repository permissions, and does not publish packages, sign artifacts, create releases or update latest.

This PR preserves develop's C++/Python 2.5.1.dev and Maven 2.5.1-SNAPSHOT versions. Version-helper success tests use a temporary release fixture so the helper tests also run in a development checkout; development-version rejection remains covered.

Validation

On this develop-based branch:

  • Packaging helper suite: 67 passed, 7 opt-in skips in the default run. All 11 tests in the native-install module passed with installation tests enabled on both macOS ARM64 and Linux ARM64.
  • Workflow contracts and shell syntax, actionlint (allowing the two existing newer runner labels), Python formatting, Git whitespace checks, and dependency-source policy tests passed.
  • Installed static SDK regressions cover C and C++ pkg-config linking, all enabled compression codecs with write/read round trips, SYSTEM LZ4, and a BUNDLED Ninja Multi-Config build. These installation tests now run in the dependency-source PR workflow.
  • Reproduced all four Linux Debug UTF-8 failures before the fix; all nine related CLI regressions passed afterwards. Full Linux ARM64 GCC 14 Debug ASan/UBSan CTest with LZMA2 enabled: 1,172 discovered, 1,159 passed, 3 conditional skips, 10 existing disabled tests, zero failures. This container build used real Git metadata and -fsigned-char to match Ubuntu x86_64 golden-output assumptions.
  • Built and installed a macOS ARM64 bundled SDK, then archived it and tested it after extraction into another directory. Mach-O dependency checks passed; 222 C++ tests passed with 3 existing disabled cases; CLI checks passed on all four fixtures; Go vet, tests and race checks passed.

Before porting the packaging commits to develop, the release-based implementation at d8b70c69b72fa7924bb29cd3995b7d9854aef983 passed the complete multi-platform native release workflow: 60/60 jobs successful. That run is evidence for the release-based implementation; it is not a run of this PR's develop-based head.

Related: #1012 (the existing rc/2.5.0 PR).

Fix MSVC language feature detection, runtime selection, native ARM64 builds, dependency downloads and relocatable library paths. Export headers, CMake and pkg-config metadata, static dependencies and native package components. Record the resolved CRT and runtime filename in build-tree and installed SDK metadata so independent consumers can select matching compiler options. Allow existing API tests to consume an installed SDK and include dependency-download regression tests. Inherit the selected compiler in isolated ANTLR4 dependency test configurations.
Build native SDKs, DEB/RPM packages, Windows ZIPs, Homebrew bottles and Python 3.9-3.14 wheels. Reuse SDKs and automatically read their MSVC runtime configuration. Ship runtime-only wheels with one native library, stripped Unix binaries and license texts; reject build-only files, duplicate libraries and ELF debug sections. Test every installed package, wheel and relocated SDK in fresh jobs using existing C++ and Go tests, CLI operations on Python resources and the full pytest suite. Gate checksummed bundle assembly on all test matrices and retire the separate wheel workflow.
@hongzhi-gao

Copy link
Copy Markdown
Contributor
  1. cpp/cmake/libtsfile.pc.in:30

    • Trigger: Install a static TsFile build with codecs enabled and link a consumer using the installed pkg-config metadata.
    • Potential issue: The metadata supplies only -ltsfile and has no Libs.private or Requires.private. Static archives do not absorb codec dependencies; the new CMake integration explicitly installs and exports those dependencies separately. Consequently, even pkg-config's static mode omits the required codec archives and platform libraries, leaving unresolved symbols when linking consumers.
  2. cpp/tools/CMakeLists.txt:72

    • Trigger: Build a shared Unix configuration with CMAKE_INSTALL_LIBDIR=lib64 (or a multiarch library directory), then execute the build-tree CLI without a library-path override.
    • Potential issue: BUILD_WITH_INSTALL_RPATH forces the build-tree executable to search the installation layout. The CLI is built under build/bin and TsFile under build/lib, but its runtime search path becomes build/lib64 or the configured multiarch directory. This replaces CMake's working build-tree RPATH and prevents the CLI from loading the freshly built library.

Include bundled and system link dependencies in static pkg-config metadata, with per-configuration archive paths and C++ runtime flags. Preserve CMake's build-tree CLI RPATH while keeping relocated installations working in lib64 and multiarch layouts.

Align exported path-parser macros with the library's object targets to resolve Linux Debug UTF-8 failures. Exercise installed C and C++ consumers with real codec round trips and run the installation regressions in CI.
@ColinLeeo

Copy link
Copy Markdown
Contributor Author

Thanks @hongzhi-gao for the review. Both points in your comment are addressed in 81f8b8f.

  1. Static pkg-config linkage: Added Libs.private with the private codec archives, platform libraries and C++ runtime dependencies. Bundled archive paths are generated per configuration and remain relative to the installed SDK. The regression tests link both C and C++ consumers using static pkg-config metadata, then perform real write/read round trips with every enabled compression codec. They cover relocated BUNDLED SDKs with Ninja Multi-Config and SYSTEM LZ4.

  2. CLI RPATH: Removed BUILD_WITH_INSTALL_RPATH, allowing CMake to retain the correct build-tree RPATH. The installed RPATH is calculated relative to the installed bindir and libdir. New tests run the build-tree CLI and the relocated installed CLI for both lib64 and multiarch layouts, with library-path overrides cleared and the build directory deleted before testing the installed CLI.

All 11 native-install tests passed on both macOS ARM64 and Linux ARM64. These installation regressions now also run in the dependency-source PR workflow. The new PR CI run is still in progress, with no failed checks so far.

@ColinLeeo
ColinLeeo requested a review from hongzhi-gao October 16, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants