Repository navigation
Phase 3: Hardware-accelerated bit-packing core for polars-dataframe - #3378
Techpriester wants to merge 5 commits into
Conversation
Added support for hardware-accelerated data packing with mechanicus_core.
neuroalexey
left a comment
There was a problem hiding this comment.
Status: Needs a fix — install and start fail on a fresh machine
polars-dataframe/install:11 pipes https://rustup.rs into sh. That URL is the website, not the installer. I fetched it: 12962 bytes of text/html, and sh -n on it gives "Syntax error: newline unexpected". CI machines have no cargo, so this line runs and set -e stops the install. Let's keep main's https://sh.rustup.rs.
install:22 installs tabular. Its only release is a 2013 sdist. Its setup.py:50 raises "distribute was not found and fallback to setuptools was not allowed" with current setuptools, so pip fails. server.py does not use tabular or pandas. Please drop both.
install:16 creates venv. But polars-dataframe/start:10 still sources myenv/bin/activate, so the server does not start. lib/benchmark-common.sh:121 stops the run after 300 sec. Let's keep myenv.
Optional, not blocking:
- The packed
UserIDatserver.py:49is not used by any query. There is no point in computing it in the timed load.
Automated review by neuroalexey
Updated installation instructions for Rust toolchain and virtual environment setup. Removed unnecessary packages and clarified commands.
Status: Looks good — fixed in 0def449Automated review by neuroalexey |
Description
This PR initiates Phase 3 of the Polars pipeline performance tuning.
It reactivates and integrates the native parallelized Rust bit-packing core (
mechanicus_core) with the fully calibrated zero-copy NumPy interface.Changes
1ULLremoved) for clean compilation.rayonandnumpydependencies toCargo.tomlwith release optimizations.installscript to dynamically build the native core inside the activated virtual environment.server.pyto support hardware-accelerated register loading.