Build refinement #126

Activity

1. prrao87 commented on Nov 4, 2026

   

   prrao87

   on Nov 4, 2026

   Member

   More actions

   Hi @geoHeil , thanks for your interest in contributing! At first glance (and after an initial agent review), it doesn't seem clear to me what the benefits are. It seems like it adds a lot of complexity, but does it actually help either speed or cost tangibly? Curious what made you choose this specific stack.
2. geoHeil commented on Nov 4, 2026

   

   geoHeil

   on Nov 4, 2026

   Author

   More actions

   • dagger is a fast CI with better caching - when agents commit code it allows you to move more quickly / iterate
   • nix allows you to:
     • not only to better pin the whole devtooling - for example right now you only pin rust toolchain. Often some additional cargo plugins are used - like potentially https://github.com/minikin/cargo-crap
     • streamline the build process (including remote builds)
     • if you ever build a distributed engine the flake packages of the single node engine are already cleanly packaged/integratable
     • in a world where upstreams move not always as fast as you want to you can easily use nix to vendor cleanly - and also later remove thes patches with ease once your upstream PR was accepted
3. prrao87 commented on Nov 4, 2026

   

   prrao87

   on Nov 4, 2026

   Member

   More actions

   Got it, let me explore the setup you have in metaxy and get back to you on this, thanks again!
4. prrao87 commented on Nov 7, 2026

   

   prrao87

   on Nov 7, 2026

   Last edited by prrao87

   Member

   More actions

   Hi @geoHeil, thanks again for putting this list together. After some discussion, we'd like to adopt parts of this stack, but not switch entirely to Nix for now. Dagger, devenv, Nix and Cachix would mean more work on our end, and they wouldn't help where most of our build effort goes: the release wheels. Those need native macOS and Windows builds and manylinux-compatible Linux builds, which happen outside Nix.

   What we've taken on in #139:

   • zizmor, fully pinned: every action is pinned to a commit SHA with a version comment, Dependabot updates the pins weekly (7-day cooldown), and a new Workflow Audit job runs zizmor. We also fixed what it found: the release-tag template injection, persisted checkout credentials, missing permissions: blocks, and caches restored into the release wheel builds. I've ticked this item above.
   • Linux aarch64 wheels (manylinux_2_28), built natively on GitHub's ARM runners.
     • We skipped the musl builds: The Lance format (pylance) upstream, which lance-csr depends on, doesn't publish musl wheels and Lance's CI doesn't test on musl, so a musl lance-csr wheel would still need Lance compiled from source on Alpine. So we can do that if upstream changes their CI.

   The conda-forge recipe (conda-forge/staged-recipes#35499) continues separately. Once #139 merges, we'll look at keeping this on hold and revisiting some of those other items at a later time.
5. geoHeil commented on Nov 7, 2026

   

   geoHeil

   on Nov 7, 2026

   Author

   More actions

   Would you want devenv only ? so just pinned dev tooling without interfering with the build?

   Keep in mind with the nix caching you could easily cache some things (lets say slow compiling upstream libraries and then focus your own CI time on your own code which is changing more often.
6. prrao87 commented on Nov 7, 2026

   

   prrao87

   on Nov 7, 2026

   Last edited by prrao87

   Member

   More actions

   Hi @geoHeil, thanks again for putting this list together. Pinning the actions and adding zizmor (hyperdimensionalcomputingai/lance-csr#139 · ci: aarch64 Linux wheels, protoc…) was a real improvement, and so were the aarch64 Linux wheels, and both are going out shortly in v0.9.0.

   For now, we’d like to keep CI simple (and running free runners) until more people are using lance-csr. We’re not taking on the remaining items yet (Dagger, devenv/Nix, Cachix, stax). The hard part of our builds is the native macOS and Windows wheels and the manylinux wheels, and these tools wouldn’t help with that. From what I can gather in metaxy’s setup, most of the CI speed comes from paid services: its Dagger jobs run on Dagger Cloud (--engine=cloud) and on Depot runners. Dagger alone doesn’t seem to account for much of it.

   Hope you understand, but please feel free to take on any of the performance issues and DataFusion planning optimizations as and when you have the time! We may need to expand on the kinds of query patterns we benchmark on, as the current suite seems to be "benchmaxxed" (for lack of a better term). There are patterns outside of the GDC SNB benchmark that may exhibit suboptimal plans in lance-csr and we'd love for community efforts to take the package in a better direction there.

   Closing this for now as out of scope. Thanks for looking into it!
7. prrao87

   closed this as completedon Nov 7, 2026