Skip to content

Phase 3: Hardware-accelerated bit-packing core for polars-dataframe - #3378

Open
Techpriester wants to merge 5 commits into
ClickHouse:mainfrom
Techpriester:feat-mechanicus-avx512
Open

Techpriester wants to merge 5 commits into
ClickHouse:mainfrom
Techpriester:feat-mechanicus-avx512

Conversation

@Techpriester

Copy link
Copy Markdown
Contributor

Description

This PR initiates Phase 3 of the Polars pipeline performance tuning.
It reactivates and integrates the native parallelized Rust bit-packing core (mechanicus_core) with the fully calibrated zero-copy NumPy interface.

Changes

  • Fixed Rust syntax and literals (1ULL removed) for clean compilation.
  • Added rayon and numpy dependencies to Cargo.toml with release optimizations.
  • Enhanced install script to dynamically build the native core inside the activated virtual environment.
  • Updated server.py to support hardware-accelerated register loading.

@neuroalexey neuroalexey left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Status: Needs a fix — install and start fail on a fresh machine

polars-dataframe/install:11 pipes https://rustup.rs into sh. That URL is the website, not the installer. I fetched it: 12962 bytes of text/html, and sh -n on it gives "Syntax error: newline unexpected". CI machines have no cargo, so this line runs and set -e stops the install. Let's keep main's https://sh.rustup.rs.

install:22 installs tabular. Its only release is a 2013 sdist. Its setup.py:50 raises "distribute was not found and fallback to setuptools was not allowed" with current setuptools, so pip fails. server.py does not use tabular or pandas. Please drop both.

install:16 creates venv. But polars-dataframe/start:10 still sources myenv/bin/activate, so the server does not start. lib/benchmark-common.sh:121 stops the run after 300 sec. Let's keep myenv.

Optional, not blocking:

  • The packed UserID at server.py:49 is not used by any query. There is no point in computing it in the timed load.

Automated review by neuroalexey

Updated installation instructions for Rust toolchain and virtual environment setup. Removed unnecessary packages and clarified commands.
@neuroalexey

Copy link
Copy Markdown

Status: Looks good — fixed in 0def449

Automated review by neuroalexey

This branch is waiting to be deployed

1 waiting deployment
benchmark-approval — 0def4490 Waiting Nov 16, 2026 by Techpriester via launch #1315
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants